A model evaluation method and apparatus

Abstract

The application provides a model evaluation method and device, the method comprising: identifying a preset verification set according to a sensitive word recognition model to obtain the number of true positive examples and the number of true negative examples corresponding to the sensitive word recognition model; determining a target positive example accuracy rate according to the number of true positive examples and the total number of positive examples of the verification set, and determining a target negative example accuracy rate according to the number of true negative examples and the total number of negative examples of the verification set; and performing weighted calculation on the target positive example accuracy rate and the target negative example accuracy rate to obtain evaluation information. The model evaluation method provided by the application can obtain evaluation information representing the recognition accuracy of the model by performing weighted calculation on the target positive example accuracy rate and the target negative example accuracy rate. Since the evaluation information comprehensively considers the accuracy rates of the model in positive example recognition and negative example recognition, the evaluation information can more accurately indicate the recognition accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and in particular to a model evaluation method and device. BACKGROUND

[0002] In recent years, mobile Internet devices represented by smart phones and tablet computers are becoming a convenient channel for people to access the Internet and obtain information anytime and anywhere. With the rapid development of mobile Internet, network information is increasingly mixed with illegal content.

[0003] At present, related enterprises generally use a sensitive word recognition model to monitor network information to identify and process illegal content mixed in the network information and maintain a harmonious network environment.

[0004] In related technologies, the recognition accuracy of a sensitive word recognition model is usually evaluated based on the F value corresponding to the sensitive word recognition model. This evaluation method only considers the positive examples (i.e., corpus recognized by the model as not containing sensitive words) recognized by the sensitive word recognition model, which may result in a large difference between the model recognition accuracy indicated by the evaluation result and the actual model recognition accuracy. That is, when the recognition accuracy of a sensitive word recognition model is evaluated based on related technologies, the reliability of the obtained evaluation result is poor. SUMMARY

[0005] Embodiments of the present application provide a model evaluation method and device to solve the problem of unreliable evaluation results when evaluating the recognition accuracy of a sensitive word recognition model in related technologies.

[0006] To solve the above technical problems, the present application is implemented as follows:

[0007] In a first aspect, the embodiments of the present application provide a model evaluation method. The method comprises:

[0008] Each sample in a pre-set verification sample set is recognized by a sensitive word recognition model to obtain a recognition result corresponding to each sample in the verification sample set; wherein the verification sample set includes a plurality of positive example samples and a plurality of negative example samples, the positive example samples are samples that do not contain sensitive words, and the negative example samples are samples that include sensitive words;

[0009] The number of true positive examples and the number of true negative examples in the verification sample set are determined according to the recognition result corresponding to each sample in the verification sample set; wherein the number of true positive examples is the number of positive example samples in the verification sample set that are recognized as positive examples, and the number of true negative examples is the number of negative example samples in the verification sample set that are recognized as negative examples;

[0010] determine a target positive example accuracy according to the number of true positive examples and a total number of positive examples, and determine a target negative example accuracy according to the number of true negative examples and a total number of negative examples, wherein the total number of positive examples is a total number of positive example samples in the verification sample set, the target positive example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the positive example samples in the verification sample set, the total number of negative examples is a total number of negative example samples in the verification sample set, and the target negative example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the negative example samples in the verification sample set;

[0011] perform weighted calculation on the target positive example accuracy and the target negative example accuracy to obtain evaluation information used to represent an identification accuracy of the sensitive word identification model.

[0012] In a second aspect, the embodiments of the present application further provide a model evaluation device. The model evaluation device comprises:

[0013] a model verification module, configured to identify each sample in a pre-set verification sample set by using a sensitive word identification model to obtain an identification result corresponding to each sample in the verification sample set; wherein the verification sample set comprises a plurality of positive example samples and a plurality of negative example samples, the positive example samples are samples not containing sensitive words, and the negative example samples are samples containing sensitive words;

[0014] a statistics module, configured to determine a number of true positive examples and a number of true negative examples in the verification sample set according to the identification result corresponding to each sample in the verification sample set; wherein the number of true positive examples is a number of positive example samples identified as positive examples in the verification sample set, and the number of true negative examples is a number of negative example samples identified as negative examples in the verification sample set;

[0015] a processing module, configured to determine a target positive example accuracy according to the number of true positive examples and a total number of positive examples, and determine a target negative example accuracy according to the number of true negative examples and a total number of negative examples, wherein the total number of positive examples is a total number of positive example samples in the verification sample set, the target positive example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the positive example samples in the verification sample set, the total number of negative examples is a total number of negative example samples in the verification sample set, and the target negative example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the negative example samples in the verification sample set;

[0016] a model evaluation module, configured to perform weighted calculation on the target positive example accuracy and the target negative example accuracy to obtain evaluation information used to represent an identification accuracy of the sensitive word identification model.

[0017] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the model evaluation method described above are implemented.

[0018] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the model evaluation method described above are implemented.

[0019] In the embodiments of the present application, the number of true examples and the number of true negative examples identified by the sensitive word identification model in the verification process are obtained, and the target positive example accuracy and the target negative example accuracy of the model in the verification process are determined according to the number of true examples and the number of true negative examples, respectively. Then, the evaluation information representing the identification accuracy of the model is obtained by weighted calculation of the target positive example accuracy and the target negative example accuracy. Since the evaluation information comprehensively considers the accuracy of the model in positive example identification and negative example identification, it can more accurately indicate the identification accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a flowchart of the model evaluation method provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of the model evaluation method provided by another embodiment of the present application;

[0023] Figure 3 is a structural diagram of the model evaluation device provided by an embodiment of the present application;

[0024] Figure 4 is a structural diagram of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0026] The embodiment of the present application provides a model evaluation method. Referring to Figure 1 , Figure 1 The embodiment of the present application provides a model evaluation method. Referring to Figure 1 The embodiment of the present application provides a model evaluation method. Referring to

[0027] Step 101, using a sensitive word identification model to identify each sample in a pre-set verification sample set respectively, to obtain an identification result corresponding to each sample in the verification sample set.

[0028] The verification sample set includes a plurality of positive example samples and a plurality of negative example samples, the positive example samples are samples without sensitive words, and the negative example samples are samples including sensitive words.

[0029] The sensitive word identification model can be understood as a model obtained by training an initial model by using a training set, the initial model can be a neural network model, or a model implemented based on a sensitive word detection algorithm (for example, a Deterministic Finite Automaton (DFA) algorithm). The training set includes a plurality of positive example corpora and a plurality of negative example corpora, and the training set is derived from corpora collected in a target business scenario, and the target business scenario is an application scenario of the sensitive word identification model (which can be understood as: the sensitive word identification model can be applied to a product in the target business scenario).

[0030] For example, the target business scenario can be a yellow anti-scene, at this time, the sensitive word identification model can be applied to a yellow anti-system in the yellow anti-scene, after the corpus is input into the yellow anti-system, the sensitive word identification model can identify the input corpus, and in the case that the input corpus includes sensitive words, the publication of the input corpus is prevented.

[0031] For example, the specific product can be understood as a product produced by an application party of the sensitive word identification model, or a product produced by a partner of the application party of the sensitive word identification model.

[0032] It should be noted that the verification sample set is also derived from corpora collected in the target business scenario.

[0033] The sensitive word recognition model can be understood as inputting a plurality of samples included in the verification sample set into the sensitive word recognition model, and correspondingly obtaining the recognition result output by the sensitive word recognition model for each sample in the verification sample set (i.e., identifying the sample as a corpus sample including a sensitive word, or identifying the corpus as a corpus sample not including a sensitive word). If a sample is labeled as a corpus sample not including a sensitive word in the verification sample set, and the corpus is also identified as a corpus sample not including a sensitive word by the model, the corpus is called a true positive (TP).

[0034] Correspondingly, if a corpus is labeled as a corpus sample not including a sensitive word in the verification sample set, and the corpus is identified as a corpus sample including a sensitive word by the model, the corpus is called a false negative (FN); if a corpus is labeled as a corpus sample including a sensitive word in the verification sample set, and the corpus is identified as a corpus sample not including a sensitive word by the model, the corpus is called a false positive (FP).

[0035] Step 102, determining the number of true positives and the number of true negatives in the verification sample set according to the recognition result corresponding to each sample in the verification sample set.

[0036] The number of true positives is the number of positive samples identified as positive samples in the verification sample set, and the number of true negatives is the number of negative samples identified as negative samples in the verification sample set.

[0037] The number of true positives can be obtained by counting the number of TPs in the recognition result corresponding to each sample in the verification sample set; similarly, the number of true negatives can be obtained by counting the number of TPs in the recognition result corresponding to each sample in the verification sample set.

[0038] Step 103, determining a target positive accuracy according to the number of true positives and the total number of positives, and determining a target negative accuracy according to the number of true negatives and the total number of negatives.

[0039] The total number of positives is the total number of positive samples in the verification sample set, and the target positive accuracy is used to indicate the recognition accuracy of the sensitive word recognition model for the positive samples in the verification sample set. The total number of negatives is the total number of negative samples in the verification sample set, and the target negative accuracy is used to indicate the recognition accuracy of the sensitive word recognition model for the negative samples in the verification sample set.

[0040] In an example, a ratio of the number of true positives to a total number of positive examples in the validation sample set can be determined as a target positive example accuracy, i.e., target positive example accuracy = number of true positives ÷ total number of positive examples in the validation sample set x 100% (for example, if the number of true positives is 99 and the total number of positive examples in the validation sample set is 100, the target positive example accuracy is 99%). The total number of positive examples in the validation sample set can be understood as a total number of positive example samples included in the validation sample set, and can be obtained by counting the number of corpus samples in the validation sample set that are marked as not including sensitive words, or by summing the number of true positives and the number of false negatives output by the sensitive word identification model for the validation sample set (i.e., total number of positive examples in the validation sample set = number of TPs + number of FNs).

[0041] Similarly, a ratio of the number of true negatives to a total number of negative examples in the validation sample set can be determined as a target negative example accuracy, i.e., target negative example accuracy = number of true negatives ÷ total number of negative examples in the validation sample set x 100% (for example, if the number of true negatives is 7 and the total number of negative examples in the validation sample set is 10, the target negative example accuracy is 70%). The total number of negative examples in the validation sample set can be understood as a total number of negative example samples included in the validation sample set, and can be obtained by counting the number of corpus samples in the validation sample set that are marked as including sensitive words, or by summing the number of true negatives and the number of false positives output by the sensitive word identification model for the validation sample set (i.e., total number of negative examples in the validation sample set = number of TNs + number of FPs).

[0042] Step 104, performing weighted calculation on the target positive example accuracy and the target negative example accuracy to obtain evaluation information for representing the identification accuracy of the sensitive word identification model.

[0043] The weighted calculation on the target positive example accuracy and the target negative example accuracy can be understood as:

[0044] The first evaluation parameter = target positive example accuracy x positive example weight value of the target positive example accuracy + target negative example accuracy x negative example weight value of the target negative example accuracy (for example, if the target positive example accuracy is 99%, the positive example weight value of the target positive example accuracy is 0.2, the target negative example accuracy is 70%, and the negative example weight value of the target negative example accuracy is 0.8, the first evaluation parameter is 75.8%). The first evaluation parameter is carried in the aforementioned evaluation information, and is used to indicate the identification accuracy of the sensitive word identification model (the value range of the first evaluation parameter is 0-100%, and the greater the value of the first evaluation parameter, the higher the identification accuracy of the sensitive word identification model).

[0045] The related art evaluates the model accuracy of the sensitive word recognition model according to the number of correctly recognized corpora and the total number of corpora of the validation sample set, that is, the second evaluation parameter = the number of correctly recognized corpora ÷ the total number of corpora of the validation sample set, wherein the second evaluation parameter is a parameter used to indicate the recognition accuracy of the sensitive word recognition model in the related art (the value range of the second evaluation parameter is also 0-100%, and the greater the value of the second evaluation parameter, the higher the recognition accuracy of the sensitive word recognition model is considered), and the number of correctly recognized corpora is the sum of the number of true positives and the number of true negatives.

[0046] For example, if the number of true positives is set to 99, the total number of positive examples of the validation sample set is 100, the number of true negatives is 7, the positive weight value of the positive example accuracy is 0.2, and the negative weight value of the negative example accuracy is 0.8, then the first evaluation parameter is 75.8%, and the second evaluation parameter is 96.4%.

[0047] As can be seen from the above, the evaluation result of the recognition accuracy of the sensitive word recognition model based on the second evaluation parameter is relatively inaccurate (the second evaluation parameter differs greatly from the target negative example accuracy, and does not meet the demand of the sensitive word recognition model in application to focus on the sensitive word recognition rate). In the embodiments of the present application, the number of true positives and the number of true negatives recognized by the sensitive word recognition model in the validation process are obtained, and the target positive example accuracy and the target negative example accuracy of the model in the validation process are determined according to the number of true positives and the number of true negatives, respectively. Then, the evaluation information representing the recognition accuracy of the model is obtained by weighted calculation of the target positive example accuracy and the target negative example accuracy. Since the evaluation information comprehensively considers the accuracy of the model in positive example recognition and negative example recognition, it can more accurately indicate the recognition accuracy of the model.

[0048] In some optional embodiments, the target positive example accuracy and the target negative example accuracy are weighted calculated to obtain evaluation information for representing the recognition accuracy of the sensitive word recognition model, comprising:

[0049] obtaining a first weight parameter corresponding to the positive example sample and a second weight parameter corresponding to the negative example sample;

[0050] determining a positive weight value of the target positive example accuracy according to the first weight parameter and a weight total number, and determining a negative weight value of the target negative example accuracy according to the second weight parameter and the weight total number, wherein the weight total number is the sum of the first weight parameter and the second weight parameter;

[0051] The product of the target positive example accuracy and the positive example weight value and the sum of the product of the target negative example accuracy and the negative example weight value are determined as the evaluation information.

[0052] The positive example weight value of the target positive example accuracy determined according to the first weight parameter and the total weight number can be understood as: positive example weight value = first weight parameter ÷ total weight number. Similarly, the negative example weight value of the target negative example accuracy determined according to the second weight parameter and the total weight number can be understood as: negative example weight value = second weight parameter ÷ total weight number.

[0053] For example, if the first weight parameter corresponding to the positive example is set to 1 and the second weight parameter corresponding to the negative example is set to 4, the total weight number is 5 (5 = 1 + 4), the positive example weight value is 0.2 (0.2 = 1 ÷ 5), and the negative example weight value is 0.8 (0.8 = 4 ÷ 5).

[0054] The calculation of the target positive example accuracy, the positive example weight value, the target negative example accuracy, and the negative example weight value can be understood as: first evaluation parameter = positive example accuracy × positive example weight value of the positive example accuracy + negative example accuracy × negative example weight value of the negative example accuracy.

[0055] The first weight parameter is used to indicate the degree of attention of the sensitive word recognition model to the positive example recognition accuracy in application, and the first weight parameter can be set by the user in advance (for example, the value of the first weight parameter can be set by 5-point system, that is, the value of the first weight parameter can be any value of 1, 2, 3, 4, 5, wherein the higher the value, the more attention is paid, 1 or 2 represents general attention, 3 represents ordinary attention, 4 represents relatively high attention, and 5 represents extremely high attention). Similarly, the second weight parameter is used to indicate the degree of attention of the sensitive word recognition model to the negative example recognition accuracy in application, and the first weight parameter can also be set by the user in advance, and the first weight parameter and the second weight parameter adopt the same point system.

[0056] It should be noted that the value determination process of the first weight parameter and the value determination process of the second weight parameter are independent of each other, that is, the sum of the first weight parameter and the second weight parameter is within a predefined value range (for example, in the case of 5-point system, the sum of the first weight parameter and the second weight parameter is between 2-10).

[0057] As described above, the value determination process of the first weight parameter and the value determination process of the second weight parameter are independent of each other, the first weight parameter can accurately indicate the importance of the positive example recognition accuracy of the sensitive word recognition model in the application, and the second weight parameter can accurately indicate the importance of the negative example recognition accuracy of the sensitive word recognition model in the application, which can improve the accuracy of the determined positive example weight value and negative example weight value, and thus further improve the accuracy of the obtained first evaluation parameter.

[0058] In some optional embodiments, the obtaining the first weight parameter corresponding to the positive example sample and the second weight parameter corresponding to the negative example sample comprises:

[0059] obtaining a priority indication parameter;

[0060] In the case where the priority indication parameter indicates that the false detection priority of the sensitive word recognition model is higher than the missed detection priority of the sensitive word recognition model, the first weight parameter is set to a first value, and the second weight parameter is set to a second value;

[0061] In the case where the priority indication parameter indicates that the false detection priority of the sensitive word recognition model is higher than the missed detection priority of the sensitive word recognition model, the first weight parameter is set to a third value, and the second weight parameter is set to a fourth value;

[0062] The first value is less than the third value, and the second value is greater than the fourth value.

[0063] The missed detection priority of the sensitive word recognition model being higher than the false detection priority of the sensitive word recognition model can be understood as that the target business scenario of the sensitive word recognition model pays more attention to the missed detection problem (compared to the false detection problem), in which case, the value of the second weight parameter is increased (i.e., the second weight parameter is set to the second value) to give more weight value to the negative example recognition accuracy in the model evaluation, so as to match the high attention demand of the target business scenario to the missed detection problem, so that the obtained first evaluation parameter can more accurately indicate the model precision of the model.

[0064] Similarly, the false detection priority of the sensitive word recognition model being higher than the missed detection priority of the sensitive word recognition model can be understood as that the target business scenario of the sensitive word recognition model pays more attention to the false detection problem (compared to the missed detection problem), in which case, the value of the first weight parameter is increased (i.e., the first weight parameter is set to the third value) to give more weight value to the positive example recognition accuracy in the model evaluation, so as to match the high attention demand of the target business scenario to the false detection problem, so that the obtained first evaluation parameter can more accurately indicate the model precision of the model.

[0065] For example, the priority indication parameter can have a value of 0 or 1. When the value of the priority indication parameter is 0, it is set that the priority indication parameter indicates that the priority of the missed detection of the sensitive word recognition model is higher than the priority of the false detection of the sensitive word recognition model. When the value of the priority indication parameter is 1, it is set that the priority indication parameter indicates that the priority of the false detection of the sensitive word recognition model is higher than the priority of the missed detection of the sensitive word recognition model.

[0066] It should be noted that the priority indication parameter can be understood as a parameter input by a user.

[0067] In some optional embodiments, a difference between the ratio of the positive example samples and the negative example samples in the verification sample set and a reference ratio is within a first preset range.

[0068] The reference ratio is a ratio of positive example samples and negative example samples in samples collected in a target business scenario, and the target business scenario is an application scenario of the sensitive word recognition model.

[0069] The difference between the ratio of the positive example samples and the negative example samples in the verification sample set and the reference ratio being within the first preset range can be understood as that the proportion of the positive examples and the negative examples in the verification sample set is close to the proportion of the positive examples and the negative examples in a plurality of corpus samples included in the target business scenario. Through the above setting, the simulation degree of the verification sample set on the plurality of corpus samples included in the target business scenario can be improved, thereby further improving the evaluation accuracy of the first evaluation parameter on the recognition accuracy of the model.

[0070] The first preset range can be adaptively determined according to actual requirements (for example, the first preset range can be set as 0-0.1). The ratio of the positive examples and the negative examples in the verification sample set can be determined by the number of the positive examples in the verification sample set and the number of the negative examples in the verification sample set (for example, if the number of the positive examples in the verification sample set is 90 and the number of the negative examples in the verification sample set is 10, the ratio of the positive examples and the negative examples in the verification sample set is 9), and the reference ratio can be determined by the ratio of the number of the positive examples in the plurality of corpora included in the target business scenario and the number of the negative examples in the plurality of corpora included in the target business scenario (for example, if the number of the positive examples in the plurality of corpora included in the target business scenario is set as 95 and the number of the negative examples in the plurality of corpora included in the target business scenario is 5, the reference ratio is 19).

[0071] In the application, the ratio of the positive examples and the negative examples in the verification sample set is preferably set to be consistent with the reference ratio, that is, the difference between the ratio of the positive examples and the negative examples in the verification sample set and the reference ratio is 0, and at this time, the simulation degree of the verification sample set is the highest, and the evaluation accuracy of the first evaluation parameter on the recognition accuracy of the model can be maximally improved.

[0072] It should be noted that, in some embodiments, the difference between the ratio of positive examples and negative examples in the foregoing training set and the reference ratio can also be limited to be within a second preset range, so that the ratio of positive examples and negative examples in the training set is close to the ratio of positive examples and negative examples in the plurality of corpora included in the target business scenario, thereby improving the simulation degree of the training set on the plurality of corpora included in the target business scenario, which can further improve the training effect of the sensitive word recognition model.

[0073] It should be emphasized that, in this embodiment, the sensitive word recognition model is used for binary classification processing of the input corpus, that is, the input corpus is recognized as a sensitive word, or the input corpus is recognized as a non-sensitive word (that is, the sensitive word recognition model in this embodiment does not further distinguish the category of the recognized sensitive word).

[0074] Referring to Figure 2 , Figure 2 is a flowchart of a model evaluation method provided by another embodiment of the present application. The difference between the embodiment of the present application and the previous embodiment is mainly that the sensitive word recognition model is used for multi-classification processing of the input corpus, that is, the sensitive word recognition model further distinguishes the category of the sensitive word in the case of recognizing the input corpus as a sensitive word (for example, recognizing the input corpus as a first category of sensitive word, or recognizing the input corpus as a second category of sensitive word, etc.); in addition, the verification sample set includes N categories of negative example samples, the N categories of negative example samples are one-to-one corresponding to N categories of sensitive words, and N is an integer greater than 1.

[0075] As Figure 2 shown, the model evaluation method provided by the embodiment of the present application includes the following steps:

[0076] Step 201, using a sensitive word recognition model to respectively recognize each sample in a pre-set verification sample set, to obtain a recognition result corresponding to each sample in the verification sample set.

[0077] Step 202, according to the recognition result corresponding to each sample in the verification sample set, determining the number of true positive examples in the verification sample set and the number of true negative examples corresponding to each category of negative example samples in the N categories of negative example samples.

[0078] Step 203, respectively according to the number of true negative examples corresponding to each category of negative example samples in the N categories of negative example samples and the total number of negative examples of each category of negative example samples, determining the negative example accuracy rate corresponding to each category of negative example samples in the N categories of negative example samples.

[0079] For example, if the number of true negative examples corresponding to the negative example samples of a certain category among the N categories of negative example samples is 8, and the total number of negative examples corresponding to the negative example samples of the category is 10, then the negative example accuracy corresponding to the negative example samples of the category is 80%.

[0080] Step 204: performing weighted calculation on the target positive example accuracy and the negative example accuracy corresponding to the negative example samples of each category among the N categories of negative example samples to obtain the evaluation information.

[0081] The weighted calculation process of step 204 can be understood as:

[0082] The first evaluation parameter = target positive example accuracy x positive example weight value + negative example accuracy corresponding to the negative example samples of the first category x negative example weight value corresponding to the negative example samples of the first category + negative example accuracy corresponding to the negative example samples of the second category x negative example weight value corresponding to the negative example samples of the second category + … + negative example accuracy corresponding to the negative example samples of the Nth category x negative example weight value corresponding to the negative example samples of the Nth category.

[0083] For example, if the target positive example accuracy is set to 100%, the positive example weight value of the target positive example accuracy is 0.5, the verification sample set includes negative example samples containing a category of sensitive words and negative example samples containing a category of sensitive words, the negative example accuracy corresponding to the negative example samples containing a category of sensitive words is 25%, the negative example weight value corresponding to the negative example samples containing a category of sensitive words is 0.4, the negative example accuracy corresponding to the negative example samples containing a category of sensitive words is 100%, and the negative example weight value corresponding to the negative example samples containing a category of sensitive words is 0.1, then the first evaluation parameter is 62.5%.

[0084] As shown above, the first evaluation parameter not only comprehensively considers the accuracy of the model in positive example recognition and negative example recognition, but also distinguishes different categories of sensitive words, so it can more accurately indicate the recognition accuracy of the model. This not only meets the demand of the sensitive word recognition model in application to focus on the recognition rate of sensitive words, but also meets the demand of the target business scenario to focus on a specific category of sensitive words (i.e., by adjusting the negative example weight value of the specific category of sensitive words that the target business scenario focuses on in the calculation process of the first evaluation parameter, the final evaluation result can be inclined to the recognition accuracy of the sensitive words of the category, so as to match the demand of the target business scenario to focus on the specific category of sensitive words.

[0085] It should be noted that each category of negative example samples among the N categories of negative example samples corresponds to a sensitive word priority and a weight value, and the sensitive word priority and the weight value corresponding to each category of negative example samples among the N categories of negative example samples are in a positive correlation.

[0086] The sensitive word priority should be understood as the degree of attention of the target business scenario to a certain category of sensitive words. The higher the value of the sensitive word priority, the higher the degree of attention of the target business scenario to the sensitive words of the category. Therefore, a higher weight value is set to reflect the high attention demand of the target business scenario to the sensitive words of the specific category.

[0087] Further, the method further comprises:

[0088] The sensitive word priority corresponding to each category of negative example samples in the N categories of negative example samples is set to a predefined initial priority, and the weight value corresponding to each category of negative example samples in the N categories of negative example samples is set to a predefined initial weight value.

[0089] In the case where the priority change request is obtained, the sensitive word priority and the weight value corresponding to the negative example sample indicated by the priority change request are updated.

[0090] It is found in the application that in the case where there are N categories of sensitive words in the target business scenario, the target business scenario usually highly pays attention to one or two categories of sensitive words among the N categories of sensitive words, and the remaining other categories of sensitive words among the N categories of sensitive words are basically consistent in the degree of attention. Therefore, compared with the way of setting the sensitive word priority and the weight value corresponding to each category of negative example samples in the N categories of negative example samples, the way of keeping the sensitive word priority and the weight value corresponding to each category of negative example samples in the N categories of negative example samples consistent first, and then updating the sensitive word priority and the weight value corresponding to at least one category of negative example samples highly paid attention to by the target business scenario through the priority change request according to the actual demand of the target business scenario, can simplify the setting steps of the sensitive word priority and the weight value corresponding to different categories of negative example samples while meeting the high attention demand of the target business scenario to the sensitive words of the specific category.

[0091] The initial priority and the initial weight value can be adaptively adjusted according to actual needs, and the specific values of the initial priority and the initial weight value are not limited in the application.

[0092] In addition, the foregoing updating operation can be understood as increasing the sensitive word priority and the weight value corresponding to the negative example sample indicated by the priority change request, or can be understood as decreasing the sensitive word priority and the weight value corresponding to the negative example sample indicated by the priority change request.

[0093] For example, in the case that the weight value is a 5-point system and the target business scenario pays high attention to the second type of sensitive words among the four types of sensitive words, the weight values corresponding to the first type of negative example samples, the second type of negative example samples, the third type of negative example samples, and the fourth type of negative example samples can be set as 3, 5, 3, and 3 respectively.

[0094] It should be noted that in this embodiment, the difference between the ratio of the positive examples and the N types of negative example samples (corresponding to N types of sensitive words respectively) in the verification sample set and the reference ratio can also be limited to be within a third preset range, and at this time, the reference ratio is the ratio of the positive examples and the N types of negative example samples in the corpus collected in the target business scenario. In this way, the simulation degree of the verification sample set on the multiple corpora included in the target business scenario is improved, thereby further improving the evaluation accuracy of the first evaluation parameter on the recognition accuracy of the model.

[0095] Similarly, in this embodiment, the difference between the ratio of the positive examples and the N types of negative example samples in the training set and the reference ratio can also be limited to be within a fourth preset range. In this way, the simulation degree of the training set on the multiple corpora included in the target business scenario is improved, thereby further improving the training effect of the sensitive word recognition model.

[0096] For example, it is assumed that the multiple corpus samples included in the target business scenario include corpus samples containing sensitive word 1, corpus samples containing sensitive word 2, and corpus samples containing only non-sensitive words (i.e., corpus samples not containing sensitive words), wherein the sensitive word priority of sensitive word 1 is higher than that of sensitive word 2, and the quantity ratio of sensitive word 1, sensitive word 2, and non-sensitive word in the multiple corpus samples included in the target business scenario is 3:1:1.

[0097] If the test set includes 100 corpora, 60 of which are marked as non-sensitive words, 20 of which are marked as sensitive word 1, and 20 of which are marked as sensitive word 2, the sensitive word recognition model is applied to recognize the 100 corpora included in the test set, and the recognition result shows that the number of corpora recognized as sensitive word 1 and also marked as sensitive word 1 in the verification sample set is 5, and the number of corpora recognized as sensitive word 2 and also marked as sensitive word 2 in the verification sample set is 20.

[0098] Based on the related art, the model evaluation is performed on the above recognition result, and TP=60, TN=25, FP=15, and FN=0 are obtained, and the evaluation result is:

[0099]

[0100] As can be seen from the above, although the sensitive word recognition model has an accuracy of 100% for the sensitive word 2, the recognition accuracy of the sensitive word recognition model for the sensitive word 1 is only 25% in the case where the target business scenario pays more attention to the sensitive word 1. Since the related art does not distinguish the categories of sensitive words when evaluating the model precision of the sensitive word recognition model, the evaluation results (i.e., the second evaluation parameter and the F value) obtained will overestimate the model precision of the sensitive word recognition model.

[0101] And in the case where the weight parameter corresponding to the positive example is 5, the weight parameter corresponding to the negative example of the sensitive word 1 is 4, and the weight parameter corresponding to the negative example of the sensitive word 2 is 1, the application of the model evaluation method provided in the present application can obtain TP = 60, TN1 = 5, TN2 = 20, FP1 = 15, FP2 = 0, FN1 = 0, and FN2 = 0 (TN1 is the number of corpus samples that are recognized as the sensitive word 1 and are also marked as the sensitive word 1 in the verification sample set, TN2 is the number of corpus samples that are recognized as the sensitive word 1 and are also marked as the sensitive word 1 in the verification sample set, FP1 is the number of corpus samples that are recognized as non-sensitive words but are marked as the sensitive word 1 in the verification sample set, FP2 is the number of corpus samples that are recognized as non-sensitive words but are marked as the sensitive word 2 in the verification sample set, FN1 is the number of corpus samples that are recognized as the sensitive word 1 but are marked as non-sensitive words in the verification sample set, and FN2 is the number of corpus samples that are recognized as the sensitive word 2 but are marked as non-sensitive words in the verification sample set), and the evaluation results obtained are:

[0102]

[0103] The first evaluation parameter = (positive example accuracy * positive example accuracy weight) + (sensitive word 1 accuracy * sensitive word 1 accuracy weight) + (sensitive word 2 accuracy * sensitive word 2 accuracy weight) = 50% + 10% + 10% = 70%

[0104] + 10% = 70%

[0105] In this example, the model evaluation method provided in the present embodiment not only distinguishes the sensitive word 1 and the sensitive word 2, but also limits the weight parameter corresponding to the sensitive word 1 to be higher than the weight parameter corresponding to the sensitive word 2, so as to meet the demand that the target business scenario pays more attention to the sensitive word 1, and finally the recognition accuracy of the sensitive word recognition model can be more accurately indicated by performing weighted calculation on the positive example accuracy, the sensitive word 1 accuracy, and the sensitive word 2 accuracy, respectively.

[0106] Referring to Figure 3 , Figure 3 is a structural diagram of the model evaluation device provided in the present embodiment. As shown in Figure 3 , the model evaluation device 300 comprises:

[0107] The model verification module 301 is configured to identify each sample in the preset verification sample set by using the sensitive word identification model, to obtain an identification result corresponding to each sample in the verification sample set; wherein the verification sample set includes a plurality of positive example samples and a plurality of negative example samples, the positive example sample is a sample without a sensitive word, and the negative example sample is a sample including a sensitive word.

[0108] The statistical module 302 is configured to determine a number of true positive examples and a number of true negative examples in the verification sample set according to the identification result corresponding to each sample in the verification sample set; wherein the number of true positive examples is a number of positive example samples in the verification sample set identified as positive examples, and the number of true negative examples is a number of negative example samples in the verification sample set identified as negative examples.

[0109] The processing module 303 is configured to determine a target positive example accuracy according to the number of true positive examples and a total number of positive examples, and determine a target negative example accuracy according to the number of true negative examples and a total number of negative examples, wherein the total number of positive examples is a total number of positive example samples in the verification sample set, the target positive example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the positive example samples in the verification sample set, the total number of negative examples is a total number of negative example samples in the verification sample set, and the target negative example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the negative example samples in the verification sample set.

[0110] The model evaluation module 304 is configured to perform weighted calculation on the target positive example accuracy and the target negative example accuracy, to obtain evaluation information used to represent an identification accuracy of the sensitive word identification model.

[0111] Optionally, the verification sample set includes N categories of negative example samples, the N categories of negative example samples correspond to N categories of sensitive words one by one, and N is an integer greater than 1.

[0112] The processing module 303 includes:

[0113] The first processing sub-module is configured to determine a negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples according to the number of true negative examples corresponding to each category of negative example samples and the total number of negative examples of each category of negative example samples; wherein the target negative example accuracy includes the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples.

[0114] The model evaluation module 304 includes:

[0115] The first evaluation submodule is configured to perform weighted calculation on the target positive example accuracy and the negative example accuracy corresponding to each of the N categories of negative example samples to obtain the evaluation information.

[0116] Optionally, each of the N categories of negative example samples corresponds to a sensitive word priority and a weight value, and the sensitive word priority and the weight value corresponding to each of the N categories of negative example samples are in a positive correlation relationship.

[0117] Optionally, the model evaluation device 300 further includes:

[0118] The weight initialization module is configured to set the sensitive word priority corresponding to each of the N categories of negative example samples to a predefined initial priority and set the weight value corresponding to each of the N categories of negative example samples to a predefined initial weight value.

[0119] The weight updating module is configured to update the sensitive word priority and the weight value corresponding to the negative example sample indicated by the priority change request in a case where the priority change request is obtained.

[0120] Optionally, a difference between the ratio of the positive example samples and the negative example samples in the verification sample set and a reference ratio is within a first preset range.

[0121] The reference ratio is a ratio of positive example samples and negative example samples in samples collected in a target business scenario, and the target business scenario is an application scenario of the sensitive word recognition model.

[0122] Optionally, the model evaluation module 304 includes:

[0123] The parameter acquisition submodule is configured to acquire a first weight parameter corresponding to the positive example sample and a second weight parameter corresponding to the negative example sample.

[0124] The weight calculation submodule is configured to determine a positive example weight value of the target positive example accuracy according to the first weight parameter and a total weight value, and determine a negative example weight value of the target negative example accuracy according to the second weight parameter and the total weight value, where the total weight value is a sum of the first weight parameter and the second weight parameter.

[0125] The weighted calculation submodule is configured to determine a sum of a product of the target positive example accuracy and the positive example weight value and a product of the target negative example accuracy and the negative example weight value as the evaluation information.

[0126] Optionally, the parameter acquisition submodule includes:

[0127] The first obtaining unit is configured to set the first weight parameter to a first value and set the second weight parameter to a second value in a case where the priority indication parameter indicates that the false negative priority of the sensitive word recognition model is higher than the false positive priority of the sensitive word recognition model.

[0128] The second obtaining unit is configured to set the first weight parameter to a third value and set the second weight parameter to a fourth value in a case where the priority indication parameter indicates that the false positive priority of the sensitive word recognition model is higher than the false negative priority of the sensitive word recognition model.

[0129] The first value is less than the third value, and the second value is greater than the fourth value.

[0130] The model evaluation apparatus 300 provided in the embodiments of the present application can implement each process in the above-described method embodiments, and thus details are not repeated here.

[0131] The model evaluation apparatus 300 provided in the embodiments of the present application can implement each process in the above-described method embodiments, and thus details are not repeated here.

[0132] Referring to Figure 4 , Figure 4 is a structural diagram of an electronic device provided in yet another embodiment of the present application, as shown in the figure, the electronic device 400 includes a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor, each component in the model evaluation apparatus 400 is coupled together through a bus interface 403, and the computer program is executed by the processor 401 to implement the following steps: Figure 4

[0133] The sensitive word recognition model is used to identify each sample in the pre-set verification sample set respectively to obtain an identification result corresponding to each sample in the verification sample set; wherein the verification sample set includes a plurality of positive samples and a plurality of negative samples, the positive sample is a sample that does not contain a sensitive word, and the negative sample is a sample that contains a sensitive word;

[0134] ​determine a number of true positive examples and a number of true negative examples in the verification sample set according to the identification result corresponding to each sample in the verification sample set, wherein the number of true positive examples is a number of positive example samples in the verification sample set identified as positive examples, and the number of true negative examples is a number of negative example samples in the verification sample set identified as negative examples;

[0135] determine a target positive example accuracy according to the number of true positive examples and a total number of positive examples, and determine a target negative example accuracy according to the number of true negative examples and a total number of negative examples, wherein the total number of positive examples is a total number of positive example samples in the verification sample set, the target positive example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the positive example samples in the verification sample set, the total number of negative examples is a total number of negative example samples in the verification sample set, and the target negative example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the negative example samples in the verification sample set;

[0136] perform weighted calculation on the target positive example accuracy and the target negative example accuracy to obtain evaluation information used to represent an identification accuracy of the sensitive word identification model.

[0137] Optionally, the verification sample set includes N categories of negative example samples, the N categories of negative example samples correspond to N categories of sensitive words one by one, and N is an integer greater than 1.

[0138] The target negative example accuracy is determined according to the number of true negative examples and the total number of negative examples.

[0139] determine a negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples according to the number of true negative examples corresponding to each category of negative example samples and the total number of negative example samples of each category of negative example samples, wherein the target negative example accuracy includes the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples.

[0140] The target positive example accuracy and the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples are weighted and calculated to obtain the evaluation information.

[0141] The target positive example accuracy and the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples are weighted and calculated to obtain the evaluation information.

[0142] Each category of negative example samples in the N categories of negative example samples corresponds to a sensitive word priority and a weight value, and the sensitive word priority and the weight value corresponding to each category of negative example samples in the N categories of negative example samples are in a positive correlation relationship.

[0143] Optionally, the computer program is further configured to, when executed by the processor 401, cause the processor 401 to:

[0144] set the sensitive word priority corresponding to each of the N categories of negative example samples to a predefined initial priority, and set the weight value corresponding to each of the N categories of negative example samples to a predefined initial weight value;

[0145] In the case where the priority change request is obtained, the sensitive word priority and the weight value corresponding to the negative example sample indicated by the priority change request are updated.

[0146] Optionally, a difference between the ratio of the positive example samples and the negative example samples in the verification sample set and a reference ratio is within a first preset range.

[0147] The reference ratio is a ratio of positive example samples and negative example samples in samples collected in a target business scenario, and the target business scenario is an application scenario of the sensitive word recognition model.

[0148] Optionally, the target positive example accuracy and the target negative example accuracy are weighted to obtain evaluation information for representing recognition accuracy of the sensitive word recognition model, including:

[0149] obtaining a first weight parameter corresponding to the positive example sample and a second weight parameter corresponding to the negative example sample;

[0150] determining a positive example weight value of the target positive example accuracy according to the first weight parameter and a total weight, and determining a negative example weight value of the target negative example accuracy according to the second weight parameter and the total weight, wherein the total weight is a sum of the first weight parameter and the second weight parameter;

[0151] determining a sum of a product of the target positive example accuracy and the positive example weight value and a product of the target negative example accuracy and the negative example weight value as the evaluation information.

[0152] Optionally, the obtaining of the first weight parameter corresponding to the positive example sample and the second weight parameter corresponding to the negative example sample includes:

[0153] obtaining a priority indication parameter;

[0154] in the case where the priority indication parameter indicates that a missed detection priority of the sensitive word recognition model is higher than a false detection priority of the sensitive word recognition model, setting the first weight parameter to a first value and setting the second weight parameter to a second value;

[0155] In a case where the priority indication parameter indicates that the false detection priority of the sensitive word recognition model is higher than the missed detection priority of the sensitive word recognition model, the first weight parameter is set to a third value, and the second weight parameter is set to a fourth value.

[0156] The first value is less than the third value, and the second value is greater than the fourth value.

[0157] It should be understood that, in the embodiments of the present application, the computer program can implement the processes in the above-mentioned model evaluation method embodiments when executed by the processor 401, and thus details are not repeated here.

[0158] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the processes in the above-mentioned model evaluation method embodiments and achieve the same technical effects. Details are not repeated here. The computer readable storage medium includes a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0159] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, such that processes, methods, articles, or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles, or devices. Without more limitations, the element defined by the phrase "including a" does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0160] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform, and of course, they can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for making a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the embodiments of the present application.

[0161] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A model evaluation method, characterized by, The method comprises the following steps: identifying each sample in a preset verification sample set by using a sensitive word identification model to obtain an identification result corresponding to each sample in the verification sample set; wherein the verification sample set comprises a plurality of positive example samples and a plurality of negative example samples, the positive example samples are samples without sensitive words, and the negative example samples are samples including sensitive words; determining the number of true positive examples and the number of true negative examples in the verification sample set according to the identification result corresponding to each sample in the verification sample set; wherein the number of true positive examples is the number of positive example samples in the verification sample set that are identified as positive examples, and the number of true negative examples is the number of negative example samples in the verification sample set that are identified as negative examples; determining a target positive example accuracy according to the number of true positive examples and the total number of positive examples, and determining a target negative example accuracy according to the number of true negative examples and the total number of negative examples, wherein the total number of positive examples is the total number of positive example samples in the verification sample set, the target positive example accuracy is used to indicate the identification accuracy of the sensitive word identification model for the positive example samples in the verification sample set, the total number of negative examples is the total number of negative example samples in the verification sample set, and the target negative example accuracy is used to indicate the identification accuracy of the sensitive word identification model for the negative example samples in the verification sample set; performing weighted calculation on the target positive example accuracy and the target negative example accuracy to obtain evaluation information for representing the identification accuracy of the sensitive word identification model.

2. The method of claim 1, wherein, The verification sample set comprises N categories of negative example samples, the N categories of negative example samples correspond to N categories of sensitive words one by one, and N is an integer greater than 1; The target negative example accuracy is determined according to the number of true negative examples and the total number of negative examples, comprising: determining the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples according to the number of true negative examples corresponding to each category of negative example samples and the total number of negative examples of each category of negative example samples; wherein the target negative example accuracy comprises the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples; The target positive example accuracy and the negative example accuracy corresponding to each category of negative example samples in the N categories of negative example samples are weighted calculated to obtain the evaluation information. Each category of negative example samples in the N categories of negative example samples corresponds to a sensitive word priority and a weight value, and the sensitive word priority and the weight value corresponding to each category of negative example samples in the N categories of negative example samples are in a positive correlation.

3. The method of claim 2, wherein, The method further comprises:

4. The method of claim 3, wherein, setting the sensitive word priority corresponding to each category of negative example samples in the N categories of negative example samples to a predefined initial priority, and setting the weight value corresponding to each category of negative example samples in the N categories of negative example samples to a predefined initial weight value. ​ In a case where the priority change request is acquired, the sensitive word priority and the weight value corresponding to the negative example sample indicated by the priority change request are updated.

5. The method of claim 1, wherein, A difference between a ratio of positive example samples and negative example samples in the verification sample set and a reference ratio is within a first preset range; The reference ratio is a ratio of positive example samples and negative example samples in samples collected in a target business scenario, and the target business scenario is an application scenario of the sensitive word recognition model.

6. The method of claim 1, wherein, The target positive example accuracy and the target negative example accuracy are weighted to obtain evaluation information for representing recognition accuracy of the sensitive word recognition model, including: A first weight parameter corresponding to the positive example sample and a second weight parameter corresponding to the negative example sample are acquired; A positive example weight value of the target positive example accuracy is determined according to the first weight parameter and a total weight, and a negative example weight value of the target negative example accuracy is determined according to the second weight parameter and the total weight, where the total weight is a sum of the first weight parameter and the second weight parameter; A product of the target positive example accuracy and the positive example weight value and a sum of a product of the target negative example accuracy and the negative example weight value are determined as the evaluation information.

7. The method of claim 6, wherein, The first weight parameter corresponding to the positive example sample and the second weight parameter corresponding to the negative example sample are acquired, including: A priority indication parameter is acquired; In a case where the priority indication parameter indicates that a false detection priority of the sensitive word recognition model is higher than a missed detection priority of the sensitive word recognition model, the first weight parameter is set to a first value, and the second weight parameter is set to a second value; In a case where the priority indication parameter indicates that the missed detection priority of the sensitive word recognition model is higher than the false detection priority of the sensitive word recognition model, the first weight parameter is set to a third value, and the second weight parameter is set to a fourth value; The first value is less than the third value, and the second value is greater than the fourth value.

8. A model evaluation apparatus characterized by comprising: Including: A model verification module is configured to use a sensitive word recognition model to recognize each sample in a preset verification sample set to obtain a recognition result corresponding to each sample in the verification sample set; the verification sample set includes a plurality of positive example samples and a plurality of negative example samples, the positive example samples are samples without sensitive words, and the negative example samples are samples including sensitive words; A statistical module is configured to determine a number of true positive examples and a number of true negative examples in the verification sample set according to the recognition result corresponding to each sample in the verification sample set; the number of true positive examples is a number of positive example samples in the verification sample set that are recognized as positive examples, and the number of true negative examples is a number of negative example samples in the verification sample set that are recognized as negative examples. The processing module is configured to determine a target positive example accuracy according to the number of true examples and a total number of positive examples, and determine a target negative example accuracy according to the number of true negative examples and a total number of negative examples, wherein the total number of positive examples is a total number of positive example samples in the verification sample set, the target positive example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the positive example samples in the verification sample set, the total number of negative examples is a total number of negative example samples in the verification sample set, and the target negative example accuracy is used to indicate an identification accuracy of the sensitive word identification model on the negative example samples in the verification sample set. The model evaluation module is configured to perform weighted calculation on the target positive example accuracy and the target negative example accuracy to obtain evaluation information used to represent an identification accuracy of the sensitive word identification model.

9. An electronic device, comprising: The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the model evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the model evaluation method according to any one of claims 1 to 7.

Images