Object labeling method, device, equipment and medium

By using cascading classification methods of low-miss-report and low-false alarm detection models in network data annotation, the problem of high labeling costs caused by uneven distribution of network data is solved, efficient and accurate object labeling is achieved, and the labeling costs are reduced and error rate.

CN114298222BActive Publication Date: 2024-11-08BEIJING TOPSEC NETWORK SECURITY TECH +2
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202111639800.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-29
Publication Date
2024-11-08
Estimated Expiration
2041-12-29

AI Technical Summary

Technical Problem

In the real business network environment, network data presents extremely uneven distribution characteristics, and the scale of abnormal samples is much smaller than that of normal samples, resulting in a lot of labor and time cost for the traditional purely manual labeling of network sample data.

Method used

The cascading classification method of low-fault detection model and low-fault detection model is adopted. Through preset sample sets and different loss functions training, the low-fault detection model is used to initially filter out abnormal objects, and then the low-fault detection model is used to perform two types. The weight coefficient of the missed error term is adjusted to improve labeling efficiency and accuracy.

Benefits of technology

It effectively reduces the missed labeling rate and mislabeling rate, reduces the workload of manual labeling, reduces the training cost of object labeling models, improves the reliability and accuracy of labeling models, and improves the efficiency of object labeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114298222B_ABST
    Figure CN114298222B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an object labeling method, device, equipment and medium. The object labeling method includes: obtaining an object to be labeled; inputting the object to be labeled into a low false negative detection model to obtain a first prediction result, the low false negative detection model is trained according to a preset sample set and a first loss function; if the first prediction result indicates that the object to be labeled is an abnormal object, inputting the object to be labeled into a low false positive detection model to obtain a second prediction result, the low false positive detection model is trained according to a preset sample set and a second loss function; if the second prediction result indicates that the object to be labeled is an abnormal object, verifying the object to be labeled, and using the verification result as the labeling result of the object to be labeled; wherein the false negative error term in the first loss function has a first weight coefficient, the false negative error term in the second loss function has a second weight coefficient, and the first weight coefficient is greater than the second weight coefficient. According to the embodiments of the present disclosure, the cost of object labeling can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an object labeling method, device, equipment and medium. Background Art

[0002] At present, artificial intelligence technology has been widely used in the field of network security, and using artificial intelligence technology to realize security detection of large-scale network data has become an important research direction in the field of network security.

[0003] In order to build an AI-based network data security detection system, the mainstream technical approach is to collect and annotate a large amount of network sample data, and use supervised learning or semi-supervised learning to train and optimize an intelligent system that meets the target performance requirements. However, in a real business network environment, network data presents extremely uneven distribution characteristics, and the scale of abnormal samples is much smaller than that of normal samples. If the network sample data is obtained through the traditional purely manual annotation of the labeled network data one by one, it will cost a lot of manpower and time. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an object labeling method, apparatus, device and medium.

[0005] In a first aspect, the present disclosure provides an object labeling method, comprising:

[0006] Obtain object data of the object to be labeled;

[0007] Inputting the object data into the low false negative detection model to obtain a first prediction result output by the low false negative detection model; wherein the low false negative detection model is trained according to a preset sample set and a first loss function;

[0008] If the first prediction result indicates that the object to be labeled is an abnormal object, the object data is input into a low false alarm detection model to obtain a second prediction result output by the low false alarm detection model; wherein the low false alarm detection model is trained according to a preset sample set and a second loss function;

[0009] If the second prediction result indicates that the object to be labeled is an abnormal object, verify the object to be labeled, and use the verification result as the labeling result of the object to be labeled;

[0010] Among them, the omission error term in the first loss function has a first weight coefficient, the omission error term in the second loss function has a second weight coefficient, the omission error term corresponds to the positive sample in the preset sample set, and the first weight coefficient is greater than the second weight coefficient.

[0011] In a second aspect, the present disclosure provides an object labeling device, comprising:

[0012] A first acquisition module is used to acquire object data of the object to be labeled;

[0013] A first prediction module is used to input object data into a low false negative detection model to obtain a first prediction result output by the low false negative detection model; wherein the low false negative detection model is trained according to a preset sample set and a first loss function;

[0014] A second prediction module is used to input the object data into a low false alarm detection model to obtain a second prediction result output by the low false alarm detection model if the first prediction result indicates that the object to be labeled is an abnormal object; wherein the low false alarm detection model is trained according to a preset sample set and a second loss function;

[0015] A first labeling module is used for verifying the object to be labeled if the second prediction result indicates that the object to be labeled is an abnormal object, and using the verification result as the labeling result of the object to be labeled;

[0016] Among them, the omission error term in the first loss function has a first weight coefficient, the omission error term in the second loss function has a second weight coefficient, the omission error term corresponds to the positive sample in the preset sample set, and the first weight coefficient is greater than the second weight coefficient.

[0017] In a third aspect, the present disclosure provides an object annotation device, including:

[0018] processor;

[0019] A memory for storing executable instructions;

[0020] The processor is used to read executable instructions from the memory and execute the executable instructions to implement the object labeling method of the first aspect.

[0021] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the object labeling method of the first aspect.

[0022] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0023] The object labeling method, apparatus, device and medium of the disclosed embodiments can, after acquiring the object data of the labeled object, use a low false positive detection model to determine a first prediction result corresponding to the object data, and when the first prediction result indicates that the object to be labeled is an abnormal object, use a low false positive detection model to determine a second prediction result corresponding to the object data, and then, when the second prediction result indicates that the object to be labeled is an abnormal object, verify the object to be labeled, and use the verification result as the labeling result of the object to be labeled. It can be seen that the low false positive detection model and the low false positive detection model in the disclosed embodiments can constitute a cascade classification model. In the process of labeling the object to be labeled, the object to be labeled can first be detected for low false positives of abnormal objects, and then the object to be labeled can be detected for low false positives of abnormal objects, thereby improving the efficiency of object labeling while reducing the false positive rate and the false positive rate, so as to improve the accuracy of object labeling. At the same time, the low underreporting detection model and the low false alarm detection model in the embodiment of the present disclosure can be trained based on the same preset sample set and different loss functions, and the weight coefficients of the underreporting error terms in different loss functions are different. This can not only reduce the workload of manual labeling, but also use a small number of labeled samples to train models that meet different needs, thereby reducing the training cost of the object labeling model and thus reducing the cost of object labeling. It can also increase the proportion of abnormal samples in manually labeled samples, thereby improving the reliability of the object labeling model and further improving the accuracy of object labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of an object labeling method provided by an embodiment of the present disclosure;

[0025] Figure 2 A flowchart of a model training method provided in an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of the structure of an object labeling device provided in an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of the structure of an object labeling device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0031] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0034] In order to solve the problems in the related art, the embodiments of the present disclosure provide an object labeling method, apparatus, device and medium.

[0035] Let's first combine Figure 1 to Figure 2 The object labeling method provided by the embodiment of the present disclosure is described.

[0036] Figure 1 A flow chart of an object labeling method provided by an embodiment of the present disclosure is shown.

[0037] In the embodiment of the present disclosure, the object labeling method can be executed by a server, and the server can be a cloud server or a server cluster or other device with storage and computing functions.

[0038] like Figure 1 As shown, the object labeling method may include the following steps.

[0039] S110: Obtain an object to be labeled.

[0040] In the embodiment of the present disclosure, the object to be labeled may be feature data of network data that needs to be labeled in large-scale network data.

[0041] Optionally, the network data may be an email, a network article, etc.

[0042] Optionally, the feature data may be data that can represent the features of the network data. Taking the network data as an email as an example, the feature data of the object to be annotated, i.e., the network data, may include the sender address of the email, the Uniform Resource Locator (URL) of the body, email attachments, communication behavior, email content, and other data that can represent the email features of multiple dimensions of the email.

[0043] S120, inputting the object to be labeled into a low false negative detection model to obtain a first prediction result output by the low false negative detection model; wherein the low false negative detection model is trained according to a preset sample set and a first loss function.

[0044] In an embodiment of the present disclosure, after the server obtains the object to be labeled, the object to be labeled can be input into a low false negative detection model to obtain a first prediction result output by the low false negative detection model.

[0045] The low false negative detection model is used to predict whether the object to be labeled is an abnormal object under the condition of meeting the low false negative requirement. The first prediction result may indicate whether the object to be labeled is an abnormal object or a non-abnormal object.

[0046] Specifically, the low missed detection model can be a detection model trained according to a preset sample set and a first loss function. The preset sample set may include feature data of a plurality of labeled sample data, and the preset sample set is training data for model training of the low missed detection model. The first loss function is a loss function for model optimization of the low missed detection model.

[0047] The sample data may be an email, a web article, etc. Moreover, the sample data and the web data have the same data type.

[0048] In the embodiment of the present disclosure, the first loss function may be any type of loss function, which is not limited herein. Optionally, the first loss function may be a cross entropy loss function.

[0049] Furthermore, the first loss function may include an omission error term and a false alarm error term. The omission error term corresponds to a positive sample in the preset sample set, that is, the omission error term is the prediction error of the positive sample. The false alarm error term corresponds to a negative sample in the preset sample set, that is, the false alarm error term is the prediction error of the negative sample.

[0050] Optionally, the omission error term in the first loss function may have a first weight coefficient.

[0051] The first weight coefficient may be a weight coefficient preset as required. For example, the first weight coefficient may be any value greater than 1.

[0052] S130. If the first prediction result indicates that the object to be labeled is an abnormal object, the object to be labeled is input into a low false alarm detection model to obtain a second detection result output by the low false alarm detection model; wherein the low false alarm detection model is trained according to a preset sample set and a second loss function.

[0053] In the disclosed embodiment, after the server obtains the first prediction result, the first prediction result can be judged. If the first prediction result indicates that the object to be labeled is a non-abnormal object, the object to be labeled is not subjected to secondary detection; if the first prediction result indicates that the object to be labeled is an abnormal object, the object to be labeled can be input into the low false positive detection model to obtain the second prediction result output by the low false positive detection model, so as to conduct secondary detection on the object.

[0054] The low false positive detection model is used to predict whether the object to be labeled is an abnormal object under the condition of meeting the low false positive requirement. The second prediction result may indicate whether the object to be labeled is an abnormal object or a non-abnormal object. Moreover, the second prediction result may be the same as the first prediction result or different from the first prediction result.

[0055] Specifically, the low false alarm detection model can be a detection model trained according to a preset sample set and a second loss function. The preset sample set used by the low false alarm detection model is the same as the preset sample set used by the low missed detection model, and the preset sample set is the training data when the low false alarm detection model is trained. The second loss function is the loss function when the low false alarm detection model is optimized.

[0056] In the embodiment of the present disclosure, the first loss function may be any type of loss function, which is not limited herein. Optionally, the first loss function may be a cross entropy loss function.

[0057] Furthermore, the second loss function may include an omission error term and a false alarm error term. The omission error term corresponds to a positive sample in the preset sample set, that is, the omission error term is the prediction error of the positive sample. The false alarm error term corresponds to a negative sample in the preset sample set, that is, the false alarm error term is the prediction error of the negative sample.

[0058] Optionally, the underreporting error term in the second loss function may have a second weight coefficient. The first weight coefficient may be greater than the second weight coefficient.

[0059] The second weight coefficient may be a weight coefficient that is pre-set as needed and is smaller than the first weight coefficient. For example, the second weight coefficient may be any value greater than 0 and less than 1.

[0060] Therefore, in the embodiment of the present disclosure, the weight ratio between the missed alarm error term and the false alarm error term can be adjusted to achieve the training of two different detection models using the same preset sample set.

[0061] S140: If the second prediction result indicates that the object to be labeled is an abnormal object, verify the object to be labeled, and use the verification result as the labeling result of the object to be labeled.

[0062] In the embodiment of the present disclosure, after the server obtains the second prediction result, the second prediction result can be judged. If the second prediction result indicates that the object to be labeled is an abnormal object, the object to be labeled can be verified and the verification result can be used as the labeling result of the object to be labeled.

[0063] In an embodiment of the present disclosure, after obtaining the object data of the object to be labeled, a low false positive detection model can be used to determine a first prediction result corresponding to the object data, and when the first prediction result indicates that the object to be labeled is an abnormal object, a low false positive detection model can be used to determine a second prediction result corresponding to the object data, and then when the second prediction result indicates that the object to be labeled is an abnormal object, the object to be labeled is verified, and the verification result is used as the labeling result of the object to be labeled. It can be seen that the low false positive detection model and the low false positive detection model in the embodiment of the present disclosure can constitute a cascade classification model. In the process of labeling the object to be labeled, the object to be labeled can first be detected for low false positives of abnormal objects, and then the object to be labeled can be detected for low false positives of abnormal objects, thereby improving the efficiency of object labeling while reducing the false positive rate and the false positive rate, so as to improve the accuracy of object labeling. At the same time, the low underreporting detection model and the low false alarm detection model in the embodiment of the present disclosure can be trained based on the same preset sample set and different loss functions, and the weight coefficients of the underreporting error terms in different loss functions are different. This can not only reduce the workload of manual labeling, but also use a small number of labeled samples to train models that meet different needs, thereby reducing the training cost of the object labeling model and thus reducing the cost of object labeling. It can also increase the proportion of abnormal samples in manually labeled samples, thereby improving the reliability of the object labeling model and further improving the accuracy of object labeling.

[0064] In some embodiments of the present disclosure, the server may output the object to be labeled to a human for verification, and the final labeling result of the object to be labeled is obtained by the human verification.

[0065] Therefore, in the embodiments of the present disclosure, the low missed alarm detection model and the low false alarm detection model can constitute a cascade classification model. In the process of labeling the object to be labeled, the object to be labeled can first be detected for low missed alarms of abnormal objects, and then the object to be labeled can be detected for low false alarms of abnormal objects. This can reduce the missed annotation rate and the false annotation rate, reduce the workload required for manual verification, and further improve the efficiency of object labeling.

[0066] In some other embodiments of the present disclosure, after S130, the object labeling method further includes: if the second prediction result indicates that the object to be labeled is a non-abnormal object, not verifying and labeling the object to be labeled.

[0067] In the disclosed embodiment, after the server obtains the second prediction result, the second prediction result may be judged. If the second prediction result indicates that the object to be labeled is a non-abnormal object, the object to be labeled is not verified or labeled.

[0068] In another embodiment of the present disclosure, in order to quickly train a low missed alarm detection model and a low false alarm detection model with higher reliability, before the server obtains the object data of the object to be labeled, it is also necessary to perform model training on the low missed alarm detection model and the low false alarm detection model.

[0069] In some embodiments of the present disclosure, a low missed alarm detection model and a low false alarm detection model can be trained by an iterative training method.

[0070] Figure 2 A flow chart of a model training method provided by an embodiment of the present disclosure is shown.

[0071] like Figure 2 As shown, before obtaining the object data of the object to be labeled, the object labeling method further includes the following steps.

[0072] S210: Obtain a preset sample set and a detection sample set.

[0073] In an embodiment of the present disclosure, the server may first obtain a preset sample set and a detection sample set, and then train a low missed detection model and a low false alarm detection model based on the obtained preset sample set and detection sample set.

[0074] In some embodiments of the present disclosure, the preset sample set may be a sample set used to perform model training on a low missed detection model and a low false alarm detection model using a gradient descent algorithm. The preset sample set may include multiple training samples, each of which may be feature data of a labeled sample data, and the feature data of the labeled sample data has a labeling result, and the labeling result is used to indicate whether the feature data of the sample data is an abnormal object.

[0075] In some embodiments of the present disclosure, the detection sample set may be a sample set used to optimize the low missed detection model and the low false alarm detection model. The detection sample set may include multiple test samples, each test sample is feature data of a labeled sample data, and the feature data of the labeled sample data has a labeling result, and the labeling result is used to indicate whether the feature data of the sample data is an abnormal object.

[0076] S220. Based on the preset sample set and the detection sample set, the preset low missed detection model is iteratively trained using the first loss function until a preset training cutoff condition is met, thereby obtaining a trained low missed detection model.

[0077] In the disclosed embodiment, during each iteration, the server can use the preset sample set and the first loss function to complete an iterative training of the preset low missed warning model, and use the detection sample set to test the preset low missed warning model obtained by this iterative training until the preset training cutoff condition is met, thereby obtaining a trained low missed warning detection model.

[0078] Furthermore, during one iterative training process, the server can randomly divide the preset sample set into a training set, a validation set, and a test set according to a preset ratio, and based on the gradient descent algorithm, use the training set, the validation set, the test set, and the first loss function to complete one iterative training of the preset low missed detection model, and obtain the low missed detection model trained in this iteration. In each iteration process, the server can use the training set, the validation set, the test set, and the first loss function to complete multiple sub-iteration training of the preset low missed detection model, and then obtain the low missed detection model trained in one iteration.

[0079] Furthermore, the server can use the preset low missed alarm model trained this time to predict the prediction results corresponding to each sample in the detection sample set, and determine whether this iteration meets the preset training cutoff condition. If so, a trained low missed alarm detection model is obtained. Otherwise, a labeling result for indicating that the feature data of the sample data is an abnormal object is added to the feature data of the sample data whose prediction result indicates that the feature data of the sample data is an abnormal object, so that the feature data of the sample data becomes the feature data of the labeled sample data, and new samples of the preset sample set are generated according to the feature data of each labeled sample data and its corresponding labeling result, the new samples are added to the preset sample set, and the next iterative training is started using the new preset sample set until the preset training cutoff condition is met to obtain a trained low missed alarm detection model.

[0080] Optionally, the training cutoff condition includes at least one of the following:

[0081] A. No new abnormal samples were generated when the model obtained from this iterative training was used to test the detection sample set.

[0082] Specifically, after this iterative training, if the prediction result of a sample indicates that it is an abnormal object when tested using the low false negative detection model, a new abnormal sample is generated, and this iterative training does not meet the training cutoff condition; after this iterative training, if the prediction results of all samples indicate that they are non-abnormal objects when tested using the low false negative detection model, no new abnormal samples are generated, and this iterative training meets the training cutoff condition.

[0083] B. The number of iterative training reaches the preset number.

[0084] Specifically, assuming that the preset number of times is M (M>1 and M is an integer), if this iterative training is the Mth iterative training, then this iterative training meets the training cutoff condition; otherwise, this iterative training does not meet the training cutoff condition.

[0085] C. The number of negative samples in the preset sample set reaches the preset number.

[0086] Specifically, after this iterative training, if it is detected that the number of negative samples in the preset sample set has reached the preset number, this iterative training meets the training cutoff condition; otherwise, this iterative training does not meet the training cutoff condition.

[0087] S230. Based on the preset sample set and the detection sample set, the preset low false alarm model is iteratively trained using the second loss function until the training cutoff condition is met to obtain a trained low false alarm detection model.

[0088] In the disclosed embodiment, during each iteration, the server can use the preset sample set and the second loss function to complete an iterative training of the preset low false alarm model, and use the detection sample set to test the preset low false alarm model obtained by this iterative training until the preset training cutoff condition is met, thereby obtaining a trained preset low false alarm model.

[0089] Furthermore, during one iterative training process, the server can randomly divide the preset sample set into a training set, a validation set, and a test set according to a preset ratio, and based on the gradient descent algorithm, use the training set, the validation set, the test set, and the second loss function to complete one iterative training of the preset low false alarm model, and obtain the preset low false alarm model trained in this iteration. In each iteration process, the server can use the training set, the validation set, the test set, and the first loss function to complete multiple sub-iterative training of the preset low false alarm model, and then obtain the preset low false alarm model trained in one iteration.

[0090] Furthermore, the server can use the preset low false alarm model trained this time to predict the prediction results corresponding to each sample in the detection sample set, and determine whether this iteration meets the preset training cutoff condition. If so, the trained preset low false alarm model is obtained. Otherwise, the feature data of the sample data whose prediction result is that the feature data of the sample data is an abnormal object is added with a labeling result for indicating that the feature data of the sample data is an abnormal object, so that the feature data of the sample data becomes the feature data of the labeled sample data, and new samples of the preset sample set are generated according to the feature data of each labeled sample data and its corresponding labeling result, the new samples are added to the preset sample set, and the next iterative training is started using the new preset sample set until the preset training cutoff condition is met to obtain the trained preset low false alarm model.

[0091] Optionally, the training cutoff condition includes at least one of the following:

[0092] A. No new abnormal samples were generated when the model obtained from this iterative training was used to test the detection sample set.

[0093] Specifically, after this iterative training, if the prediction result of a sample indicates that it is an abnormal object when tested using the preset low false alarm model, a new abnormal sample is generated, and this iterative training does not meet the training cutoff condition; after this iterative training, if the prediction results of all samples indicate that they are non-abnormal objects when tested using the preset low false alarm model, no new abnormal samples are generated, and this iterative training meets the training cutoff condition.

[0094] B. The number of iterative training reaches the preset number.

[0095] Specifically, assuming that the preset number of times is M (M>1 and M is an integer), if this iterative training is the Mth iterative training, then this iterative training meets the training cutoff condition; otherwise, this iterative training does not meet the training cutoff condition.

[0096] C. The number of negative samples in the preset sample set reaches the preset number.

[0097] Specifically, after this iterative training, if it is detected that the number of negative samples in the preset sample set has reached the preset number, this iterative training meets the training cutoff condition; otherwise, this iterative training does not meet the training cutoff condition.

[0098] In other embodiments of the present disclosure, a low missed alarm detection model and a low false alarm detection model can be obtained through one training.

[0099] Specifically, the server can first obtain a preset sample set, and then based on the preset sample set, use the first loss function to perform a model training on the preset low missed alarm model to obtain a trained low missed alarm detection model, and based on the preset sample set, use the second loss function to perform a model training on the preset low false alarm model to obtain a trained low false alarm detection model.

[0100] Among them, the method of obtaining the preset sample set and the method of one-time model iterative training are similar to the above and will not be repeated here.

[0101] Therefore, in the embodiment of the present disclosure, according to the PR curve of the deep learning model, it can be known that the false alarm rate and false alarm rate of the model have a negative correlation. Therefore, in the embodiment of the present disclosure, a low false alarm detection model and a low false alarm detection model can be trained separately, so that the low false alarm detection model and the low false alarm detection model constitute a cascade classification model, and then the two models are used to fit a model with low false alarms and low false alarms to simultaneously improve the overall accuracy and recall rate of the model. At the same time, the model training method of the embodiment of the present disclosure can quickly train a low false alarm detection model and a low false alarm detection model with higher reliability based on a preset sample set and a test sample set.

[0102] In the embodiment of the present disclosure, optionally, the step of obtaining a preset sample set in S210 may specifically include:

[0103] Obtain a target sample set that has been machine-labeled; verify each sample in the target sample set to obtain a verification result for each sample; use samples whose verification results are abnormal samples as positive samples of a preset sample set; and use samples whose verification results are non-abnormal samples as negative samples of a preset sample set.

[0104] Specifically, the server can send each sample in the target sample set that has been machine-labeled to manual verification, and network security experts will manually verify the security of the samples, and use the samples whose verification results are abnormal samples as positive samples of the preset sample set; and use the samples whose verification results are non-abnormal samples as negative samples of the preset sample set.

[0105] In the embodiment of the present disclosure, optionally, if the verification result of a certain sample is different from the machine annotation result, the sample is marked as a weighted sample. If the prediction results of the low false positive detection model and the low false positive detection model are inconsistent for a certain sample, the sample is marked as a weighted sample. If the prediction result of the low false positive detection model is inconsistent with the verification result for a certain sample, the sample is marked as a weighted sample.

[0106] Optionally, during the iterative training process, after each iterative training, when the server generates samples of a new preset sample set based on the feature data of each labeled sample data and its corresponding labeling results, the feature data of each labeled sample data can also be sent to manual verification, and the security of the sample can be manually verified by network security experts. If the result of the manual verification is a non-abnormal sample, the sample is marked as a weighted sample; if the result of the manual verification is an abnormal sample, the sample is marked as a non-weighted sample.

[0107] Optionally, during the iterative training process, after each sub-iteration training, the low false negative detection model and the low false positive detection model can be used to predict the samples in the test set respectively. If for a certain sample, the prediction results of the low false negative detection model and the low false positive detection model are inconsistent, the sample is marked as a weighted sample; otherwise, the sample is marked as a non-weighted sample.

[0108] In some other embodiments of the present disclosure, the omission error term may include a weighted omission error term and a non-weighted omission error term. The weighted omission error term may also have a third weight coefficient. The weighted omission error term is derived from positive samples belonging to a first source, and the non-weighted omission error term is derived from positive samples belonging to a second source. The second source is a source other than the first source.

[0109] Among them, the first source may include samples in the target sample set whose machine labeling results are normal samples, samples whose prediction results of the low missed alarm detection model and the low false alarm detection model are inconsistent, and samples whose prediction results of the low false alarm detection model are inconsistent with the verification results.

[0110] Furthermore, the third weight coefficient may be that the weighted sample has a greater weight than the non-weighted sample when calculating the loss function, that is, the third weight coefficient may be a coefficient greater than 1 that is pre-set as needed.

[0111] In some other embodiments of the present disclosure, the false alarm error term in the first loss function and the false alarm error term in the second loss function may respectively correspond to negative samples in a preset sample set, and the false alarm error term may include a weighted false alarm error term and a non-weighted false alarm error term. The weighted false alarm error term may also have a third weight coefficient. The weighted false alarm error term comes from positive samples belonging to a third source, and the non-weighted false alarm error term comes from positive samples belonging to a fourth source. The fourth source is a source other than the third source.

[0112] Among them, the third source may include samples in the target sample set whose machine labeling results are abnormal samples, and samples whose prediction results of the low missed detection model and the low false positive detection model are inconsistent.

[0113] Furthermore, the third weight coefficient may be that the weighted sample has a greater weight than the non-weighted sample when calculating the loss function, that is, the third weight coefficient may be a coefficient greater than 1 that is pre-set as needed.

[0114] The following takes the first loss function and the second loss function as cross entropy loss functions as an example to explain the loss function in detail:

[0115] Weighted sample omission error L jl for:

[0116]

[0117] Unweighted sample underreporting error L l for:

[0118]

[0119] Weighted sample false alarm error L jw for:

[0120]

[0121] Unweighted sample false alarm error L w for:

[0122]

[0123] The loss function L of the low missed detection model or the loss function L of the low false positive detection model is:

[0124]

[0125] Among them, when N1, N2, N3, N4 is greater than 0, 1≤i≤N1, 1≤j≤N2, 1≤k≤N3, 1≤l≤N4, when N1, N2, N3, N4 is equal to 0, the corresponding error term is 0, N=N1+N2+N3+N4, N1 is the number of weighted positive samples identified as normal samples, N2 is the number of non-weighted positive samples identified as normal samples, N3 is the number of weighted negative samples identified as abnormal samples, and N4 is the number of non-weighted negative samples identified as abnormal samples. x is the input sample data, y is the verification result, f(x) is the model prediction result, y∈{0,1}, f(x)∈[0,1], α is the false negative error weight, β is the weighted sample weight, 1<β. For low false negative detection model, α>1, for low false positive detection model, 0<α<1.

[0126] Specifically, a loss function for model optimization can be generated based on the weighted omission error term, non-weighted omission error term, weighted false alarm error term, and non-weighted false alarm error term.

[0127] The model training method provided by the embodiment of the present disclosure is described in detail below with a specific example. The low missed detection model and low false positive detection model trained by the model training method can be applied to the offline annotation of malicious email samples for large network gateways.

[0128] (1) Obtain a target sample set that has been machine-labeled. The target sample set includes 1.02 million samples of emails labeled as suspicious by the machine and 1 million samples of emails labeled as normal by the machine.

[0129] (2) The samples in the target sample set were submitted to network security experts for manual verification and labeling. For suspicious emails, a total of 1.01 million malicious emails (abnormal samples) and 10,000 safe emails (non-abnormal samples) were labeled; for normal emails, a total of 1 million safe emails (non-abnormal samples) and 0 malicious emails (abnormal samples) were labeled.

[0130] (3) Construction of labeled sample set: 1.01 million malicious emails and 1.01 million safe emails are divided into training set (606,000 malicious emails and 606,000 safe emails), validation set (202,000 malicious emails and 202,000 safe emails) and test set (202,000 malicious emails and 202,000 safe emails) according to the ratio of 6:2:2. The error term corresponding to the 10,000 emails labeled as safe samples in step (1) of the loss function is multiplied by a weight of 2, and the error term corresponding to other emails is multiplied by a weight of 1.

[0131] (4) Construction of cascade classification model: Multiply the error term corresponding to the positive sample in the loss function corresponding to the low false positive detection model by the weight 2, and use the gradient descent algorithm to train and optimize the low false positive detection model. Multiply the error term corresponding to the positive sample in the loss function corresponding to the low false positive detection model by the weight 0.5, and use the gradient descent algorithm to train and optimize the low false positive detection model.

[0132] (5) Sample label prediction and annotation: Use the low false negative detection model to predict labels for the remaining 99 million unlabeled emails (assuming that 1 million malicious emails are predicted), and use the low false positive detection model to predict classification labels for 1 million possible malicious emails (assuming that 100,000 safe emails and 900,000 malicious emails are predicted). The 900,000 malicious emails were submitted to network security experts for manual verification and annotation, and a total of 890,000 malicious emails (abnormal samples) and 10,000 safe emails (non-abnormal samples) were annotated.

[0133] (6) Put the 890,000 malicious emails and 10,000 safe emails annotated in step (5) into the annotated data set, and repeat steps (3), (4), and (5) until no new annotated malicious samples are generated.

[0134] Therefore, in the disclosed embodiments, a cascade classification model can be trained for offline annotation scenarios of massive email data at large network gateways to improve the efficiency of manual annotation of malicious email samples, reduce the proportion of missed annotations of malicious email samples, and achieve the purpose of quickly building a comprehensive annotated malicious sample data set.

[0135] In summary, in the embodiments of the present disclosure, by utilizing the cascade classification model, a combined model that satisfies both high recall and high precision can be quickly trained under the condition of a small number of samples, thereby reducing the time and cost of model training, and at the same time reducing the workload of manual labeling, thereby improving the overall performance of the combined model and reducing the probability of missing malicious samples.

[0136] In the disclosed embodiments, the labeling performance of large-scale unbalanced data is improved, the probability of sample omission and mislabeling is reduced, and the labeling result of the object to be labeled can be finally determined by using the detection model to predict the object data of the object to be labeled, which reduces the cost of object labeling, improves not only the reliability of the object labeling model, but also the accuracy and efficiency of object labeling. In addition, the disclosed embodiments will combine network security application scenarios and machine learning model training needs to solve the problem of efficient and scientific construction of the initial sample set of large-scale network data.

[0137] Combine the following Figure 3 The object labeling device provided by the embodiment of the present disclosure is described.

[0138] Figure 3 FIG. 1 is a schematic diagram showing the structure of an object labeling device provided by an embodiment of the present disclosure. Figure 3 As shown, the object labeling apparatus 300 may include a first acquisition module 310 , a first prediction module 320 , a second prediction module 330 and a first labeling module 340 .

[0139] The first acquisition module 310 can be used to acquire the object to be labeled.

[0140] The first prediction module 320 can be used to input the object to be labeled into the low false negative detection model to obtain a first prediction result output by the low false negative detection model; wherein the low false negative detection model is trained according to a preset sample set and a first loss function.

[0141] The second prediction module 330 can be used to input the object to be labeled into a low false alarm detection model if the first prediction result indicates that the object to be labeled is an abnormal object, and obtain a second prediction result output by the low false alarm detection model; wherein the low false alarm detection model is trained based on a preset sample set and a second loss function.

[0142] The first labeling module 340 can be used to verify the object to be labeled if the second prediction result indicates that the object to be labeled is an abnormal object, and use the verification result as the labeling result of the object to be labeled.

[0143] Among them, the omission error term in the first loss function has a first weight coefficient, the omission error term in the second loss function has a second weight coefficient, the omission error term corresponds to the positive sample in the preset sample set, and the first weight coefficient is greater than the second weight coefficient.

[0144] In an embodiment of the present disclosure, after obtaining the object data of the object to be labeled, a low false positive detection model can be used to determine a first prediction result corresponding to the object data, and when the first prediction result indicates that the object to be labeled is an abnormal object, a low false positive detection model can be used to determine a second prediction result corresponding to the object data, and then when the second prediction result indicates that the object to be labeled is an abnormal object, the object to be labeled is verified, and the verification result is used as the labeling result of the object to be labeled. It can be seen that the low false positive detection model and the low false positive detection model in the embodiment of the present disclosure can constitute a cascade classification model. In the process of labeling the object to be labeled, the object to be labeled can first be detected for low false positives of abnormal objects, and then the object to be labeled can be detected for low false positives of abnormal objects, thereby improving the efficiency of object labeling while reducing the false positive rate and the false positive rate, so as to improve the accuracy of object labeling. At the same time, the low underreporting detection model and the low false alarm detection model in the embodiment of the present disclosure can be trained based on the same preset sample set and different loss functions, and the weight coefficients of the underreporting error terms in different loss functions are different. This can not only reduce the workload of manual labeling, but also use a small number of labeled samples to train models that meet different needs, thereby reducing the training cost of the object labeling model and thus reducing the cost of object labeling. It can also increase the proportion of abnormal samples in manually labeled samples, thereby improving the reliability of the object labeling model and further improving the accuracy of object labeling.

[0145] In some embodiments of the present disclosure, the object labeling device 300 may further include a second labeling module, which may be used to input the object to be labeled into a low false alarm detection model to obtain a second prediction result output by the low false alarm detection model. If the second prediction result indicates that the object to be labeled is a non-abnormal object, the object to be labeled is not verified and labeled.

[0146] In some embodiments of the present disclosure, the object labeling apparatus 300 may further include a second acquisition module, a first training module, and a second training module.

[0147] The second acquisition module can be used to acquire a preset sample set and a detection sample set before acquiring the object to be labeled.

[0148] The first training module can be used to iteratively train a preset low missed detection model based on a preset sample set and a detection sample set using a first loss function until a preset training cutoff condition is met, thereby obtaining a trained low missed detection model.

[0149] The second training module can be used to iteratively train the low false alarm model based on a preset sample set and a detection sample set using a second loss function until a training cutoff condition is met to obtain a trained low false alarm detection model.

[0150] In some embodiments of the present disclosure, the second acquisition module can be specifically used to acquire a target sample set that has been machine-labeled; verify each sample in the target sample set to obtain a verification result for each sample; use samples whose verification results are abnormal samples as positive samples of a preset sample set; and use samples whose verification results are non-abnormal samples as negative samples of the preset sample set.

[0151] In some embodiments of the present disclosure, the omission error term may include a weighted omission error term and a non-weighted omission error term, the weighted omission error term further has a third weight coefficient, the weighted omission error term is derived from a positive sample belonging to a first source, and the non-weighted omission error term is derived from a positive sample belonging to a second source, and the second source is a source other than the first source;

[0152] Among them, the first source may include samples in the target sample set whose machine labeling results are normal samples, samples whose prediction results of the low missed alarm detection model and the low false alarm detection model are inconsistent, and samples whose prediction results of the low false alarm detection model are inconsistent with the verification results.

[0153] In some embodiments of the present disclosure, the false alarm error term in the first loss function and the false alarm error term in the second loss function may correspond to negative samples in a preset sample set, respectively, and the false alarm error term may include a weighted false alarm error term and a non-weighted false alarm error term, the weighted false alarm error term also has a third weight coefficient, the weighted false alarm error term is derived from positive samples belonging to a third source, and the non-weighted false alarm error term is derived from positive samples belonging to a fourth source, and the fourth source is a source other than the third source;

[0154] Among them, the third source may include samples in the target sample set whose machine labeling results are abnormal samples, and samples whose prediction results of the low missed detection model and the low false positive detection model are inconsistent.

[0155] In some embodiments of the present disclosure, the training cutoff condition may include at least one of the following:

[0156] When the model obtained by this iterative training was used to test the detection sample set, no new abnormal samples were generated;

[0157] The number of iterative training reaches the preset number;

[0158] The number of negative samples in the preset sample set reaches the preset number.

[0159] It should be noted that Figure 3 The object tagging apparatus 300 shown can execute Figure 1 to Figure 2 The various steps in the method embodiment shown in the figure are implemented Figure 1 to Figure 2 The various processes and effects in the method embodiment shown are not described in detail here.

[0160] Figure 4 A schematic structural diagram of an object labeling device provided by an embodiment of the present disclosure is shown.

[0161] In some embodiments of the present disclosure, Figure 4 The object annotation device shown may be a server, which may be a cloud server or a server cluster or other device with storage and computing functions.

[0162] like Figure 4 As shown, the object tagging device may include a processor 401 and a memory 402 storing computer program instructions.

[0163] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0164] The memory 402 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 402 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 402 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 402 is a non-volatile solid-state memory. In a particular embodiment, the memory 402 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0165] The processor 401 reads and executes the computer program instructions stored in the memory 402 to perform the steps of the object labeling method provided in the embodiment of the present disclosure.

[0166] In one example, the object annotation device may further include a transceiver 403 and a bus 404. Figure 4 As shown, the processor 401, the memory 402 and the transceiver 403 are connected via a bus 404 and communicate with each other.

[0167] The bus 404 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 404 may include one or more buses. Although embodiments of the disclosure describe and illustrate a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0168] The embodiments of the present disclosure further provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the object labeling method provided by the embodiments of the present disclosure.

[0169] The storage medium may include, for example, a memory 402 of computer program instructions, and the instructions may be executed by a processor 401 of the object annotation device to complete the object annotation method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, a non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0170] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0171] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An object labeling method, characterized in that: include: Get the object to be marked; Inputting the object to be labeled into a low false negative detection model to obtain a first prediction result output by the low false negative detection model; wherein the low false negative detection model is trained according to a preset sample set and a first loss function; If the first prediction result indicates that the object to be labeled is an abnormal object, input the object to be labeled into a low false alarm detection model to obtain a second prediction result output by the low false alarm detection model; wherein the low false alarm detection model is trained according to the preset sample set and the second loss function; If the second prediction result indicates that the object to be labeled is an abnormal object, verify the object to be labeled, and use the verification result as the labeling result of the object to be labeled; Among them, the first loss function includes an omission error term and a false alarm error term, the second loss function includes an omission error term and a false alarm error term, the omission error term in the first loss function has a first weight coefficient, the omission error term in the second loss function has a second weight coefficient, the omission error term corresponds to a positive sample in the preset sample set, and the first weight coefficient is greater than the second weight coefficient.

2. The method according to claim 1, characterized in that After inputting the object to be labeled into the low false alarm detection model to obtain a second prediction result output by the low false alarm detection model, the method further includes: If the second prediction result indicates that the object to be labeled is a non-abnormal object, the object to be labeled is not verified and labeled.

3. The method according to claim 1, characterized in that Before obtaining the object to be labeled, the method further includes: Acquire the preset sample set and the detection sample set; Based on the preset sample set and the detection sample set, iteratively training the preset low missed detection model using the first loss function until a preset training cutoff condition is met, thereby obtaining a trained low missed detection model; Based on the preset sample set and the detection sample set, the preset low false alarm model is iteratively trained using the second loss function until the training cutoff condition is met, thereby obtaining a trained low false alarm detection model.

4. The method according to claim 3, characterized in that Acquiring the preset sample set includes: Obtain a machine-labeled target sample set; Verifying each sample in the target sample set to obtain a verification result for each sample; Taking samples whose verification results are abnormal samples as positive samples of the preset sample set; The samples whose verification results are non-abnormal samples are used as negative samples of the preset sample set.

5. The method according to claim 4, characterized in that The omission error term includes a weighted omission error term and a non-weighted omission error term, the weighted omission error term further has a third weight coefficient, the weighted omission error term is derived from a positive sample belonging to a first source, and the non-weighted omission error term is derived from a positive sample belonging to a second source, and the second source is a source other than the first source; Among them, the first source includes samples in the target sample set whose machine annotation results are normal samples, samples whose prediction results of the low missed alarm detection model are inconsistent with those of the low false alarm detection model, and samples whose prediction results of the low false alarm detection model are inconsistent with verification results.

6. The method according to claim 4 or 5, characterized in that: The false alarm error term in the first loss function and the false alarm error term in the second loss function correspond to negative samples in the preset sample set respectively, the false alarm error term includes a weighted false alarm error term and a non-weighted false alarm error term, the weighted false alarm error term also has a third weight coefficient, the weighted false alarm error term comes from a positive sample belonging to a third source, and the non-weighted false alarm error term comes from a positive sample belonging to a fourth source, and the fourth source is a source other than the third source; Among them, the third source includes samples in the target sample set whose machine labeling results are abnormal samples, and samples whose prediction results of the low missed alarm detection model and the low false alarm detection model are inconsistent.

7. The method according to claim 3, characterized in that The training cutoff condition includes at least one of the following: When the model obtained by this iterative training is used to test the detection sample set, no new abnormal samples are generated; The number of iterative training reaches the preset number; The number of negative samples in the preset sample set reaches the preset number.

8. An object labeling device, characterized in that: include: A first acquisition module is used to acquire the object to be labeled; A first prediction module is used to input the object to be labeled into a low false negative detection model to obtain a first prediction result output by the low false negative detection model; wherein the low false negative detection model is trained according to a preset sample set and a first loss function; A second prediction module is used for inputting the object to be labeled into a low false alarm detection model to obtain a second prediction result output by the low false alarm detection model if the first prediction result indicates that the object to be labeled is an abnormal object; wherein the low false alarm detection model is trained according to the preset sample set and the second loss function; A first labeling module, configured to verify the object to be labeled if the second prediction result indicates that the object to be labeled is an abnormal object, and use the verification result as the labeling result of the object to be labeled; Among them, the first loss function includes an omission error term and a false alarm error term, the second loss function includes an omission error term and a false alarm error term, the omission error term in the first loss function has a first weight coefficient, the omission error term in the second loss function has a second weight coefficient, the omission error term corresponds to a positive sample in the preset sample set, and the first weight coefficient is greater than the second weight coefficient.

9. An object labeling device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the object labeling method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the object labeling method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Deep convolutional neural network-based stockyard smoke monitoring and online model updating method

    CN107025443A

  • Application layer malicious request detection method and system based on cascaded deep neural network

    CN111327608A

  • Defense method and device against sample attacks and electronic equipment

    CN111783085A

  • Code detection method and device, storage medium and terminal

    CN112307477A

  • Abnormal behavior detection method and device, computer equipment and storage medium

    CN113627406A