Processing marker data in machine learning operations

Abstract

A system, method, and software for processing tagged data in machine learning operations may be used to determine whether to re-tag tagged data. In some aspects, a method includes obtaining, by an electronic device, a set of tag data, where each of the tag data includes a feature vector and a tag; for each piece of tagged data in the tagged data set: processing the tagged data by using a plurality of machine learning models to obtain a plurality of classification results, wherein each classification result in the plurality of classification results is obtained by processing feature vectors of the tagged data by using different machine learning models in the plurality of machine learning models; and determining a tag uncertainty score for the tag data based on a difference between the average entropy score and the adjustment score; and determining whether to re-mark the one or more marked data in the marked data set based on the marked uncertainty score.

Description

Technical Field

[0001] The present disclosure relates to processing labeled data in machine learning operations. Background Art

[0002] In some implementations, machine learning is used to classify input data. For example, a machine learning model can be used for malware detection to classify whether software code may contain malware code and thereby cause a security risk. Machine learning operations can also be used for imaging processing to classify the content of an image, or for speech recognition to classify the text of audio data. Machine learning models can be used for other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a schematic diagram showing an example system that performs a re-marking determination operation according to an implementation.

[0004] Figure 2 is a flow chart showing an example process for processing labeled data according to an implementation.

[0005] Figure 3 A high-level architectural block diagram of a computer according to an implementation is shown.

[0006] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0007] In some implementations, the machine learning model is trained using labeled data. The labeled data includes a label indicating a predetermined classification result. During the training process, the machine learning model processes the labeled data to adjust the parameters of the machine learning model.

[0008] Therefore, a collection of high-quality training data is important for developing machine learning models with advanced performance. On the other hand, inaccurately labeled data may lead to biased models, resulting in inaccurate predictions and negative impacts on product and user experience.

[0009] There are two sources of uncertainty in machine learning predictions: data uncertainty and knowledge uncertainty. Data uncertainty (sometimes also called symbolic uncertainty) refers to the uncertainty or variability in the data itself. It can be due to factors such as measurement error, missing values, or outliers. On the other hand, knowledge uncertainty (also called epistemic uncertainty) arises when the model lacks understanding or knowledge about the input data. Data uncertainty is inherent in the data and cannot usually be reduced by collecting more data, while knowledge uncertainty can be reduced by providing more training data to the model.

[0010] Active learning techniques can be used to separate these two sources of uncertainty and leverage knowledge uncertainty to prioritize unlabeled data. Labeling this prioritized data will provide the model with the most useful information, which will help it make more accurate predictions. However, data uncertainty is underutilized and largely unexplored.

[0011] One of the main causes of data uncertainty is labeling noise. Therefore, one way to make full use of data uncertainty is to detect mislabeled data and improve the quality of machine learning training data by using data uncertainty. In some cases, committee query (QBC) can be used to quantify knowledge uncertainty in active learning. QBC involves training multiple models or committees on a labeled training set. When committee members disagree on the classification of a data point, it indicates a high degree of knowledge uncertainty.

[0012] In some implementations, QBC can also be used to estimate data uncertainty by considering the degree of inconsistency and the confidence of the classification. For example, in the context of binary classification, machine learning models often output probabilities ranging from 0 to 1 to represent their confidence in assigning a data point to the positive class. In some cases, a probability of 0 indicates a high confidence in the negative class, while a probability of 1 indicates a high confidence in the positive class. When a model or committee member outputs a probability of 0.5, such a probability value would indicate uncertainty or lack of confidence in classifying a data point as a positive or negative class.

[0013] In some implementations, an automatic algorithm can be used to verify labeling accuracy and select labeled data with a high likelihood of being mislabeled. A training data set of labeled data is used to train multiple machine learning models. The labeled data in the training data set may include mislabeled data. For each labeled data, a labeling uncertainty score can be determined based on the classification results produced by multiple machine learning models. The labeling uncertainty score can be used to determine whether the labeled data is likely to be mislabeled. Mislabeled data can be corrected by automatic labeling correction or submitted to a domain expert for further review. Figure 1-3 and associated descriptions provide additional details of these implementations.

[0014] The techniques described herein produce one or more technical effects. For example, the method improves the accuracy of labeled data used to train machine learning models and thus improves the performance of machine learning operations.

[0015] Figure 1 is a schematic diagram showing an example system 100 that performs a re-marking determination operation according to an implementation. At a high level, the example system 100 includes a server 120 communicatively coupled to a network 110.

[0016] Example system 100 includes network 110. Network 110 represents an application, application set, software, software module, hardware, or a combination thereof that can be configured to transmit data messages between entities in example system 100. Network 110 may include a wireless network, a wired network, the Internet, or a combination thereof. For example, network 110 may include one or more radio access networks (RANs), a core network (CN), and the Internet. RAN may include one or more radio access technologies. In some implementations, the radio access technology may be a global system for mobile communications (GSM), transitional standard 95 (IS-95), a universal mobile telecommunications system (UMTS), CDMA2000 (code division multiple access), an evolved universal mobile telecommunications system (E-UMTS), long term evaluation (LTE), advanced LTE, a fifth generation (5G), or any other radio access technology. In some cases, the core network may be an evolved packet core (EPC). RAN is a part of a wireless telecommunications system that implements radio access technologies such as UMTS, CDMA2000, 3GPP LTE, 3GPP LTE-A, and 5G.

[0017] The server 120 includes a re-labeling analyzer 104. The re-labeling analyzer 104 represents an application, a set of applications, software, a software module, hardware, or any combination thereof that is executed on the server 120 to determine whether to re-label the labeled data. In an example operation, the re-labeling analyzer 104 obtains the labeled data 140 and processes the labeled data 140 through a plurality of machine learning models (e.g., machine learning models 122, 124, and 126). Each of the machine learning models generates a classification result of the labeled data 140 and sends the classification result to the labeling uncertainty calculation module 130. The labeling uncertainty calculation module 130 calculates a labeling uncertainty score based on the classification result. The labeling uncertainty calculation module 130 determines whether to re-label the labeled data 140 based on the labeling uncertainty score. Figure 2 and associated descriptions provide additional details of these implementations.

[0018] The server 120 may be implemented using one or more computers, computer servers, or cloud computing platforms.

[0019] In some cases, server 120 may be a software service platform that detects whether the software code is malicious. In these cases, the tag data may include a feature vector and a tag of the software code, the tag indicating whether the software code is benign or malicious.

[0020] Software code can be either source code or binary code. During the software development process, source code can be created by a programmer using a text editor or visual programming tool before compilation. Source code can be developed using a human-readable programming language and can be saved in a text file. Source code can be converted by an assembler or compiler into binary software code that can be executed by a computer.

[0021] Binary software code may include a byte stream generated by compiling source code. Therefore, the binary software code may not be in a human-readable format and may not be easily parsed or analyzed by humans.

[0022] Binary software code can be a configuration of object code, executable code or byte code. Object code is the product of a compiler output of a sequence of statements or instructions in a computer language. Source code can be logically divided into multiple source files. Each source file is independently compiled into a corresponding target file including the object code. The object code in the object file is binary machine code, but they may not be ready to be executed. The object file may include incomplete references to subroutines and placeholder addresses external to it. During the linking process, these object files can be linked together to form an executable file including executable code that can be executed on a computing device. During the linking process, the linker can read the object files, resolve the references between them, perform the final code layout in the memory that determines the addresses for the code boxes and data blocks, fix the placeholder addresses using real addresses, and write out the executable file containing the executable code.

[0023] Bytecode, also referred to as portable code or p-code, is in the form of an instruction set designed for effective execution by a software interpreter. Bytecode includes compact digital code, constants and references (usually digital addresses), which encode the result of the semantic analysis of compiler parsing and execution of things such as the type, scope and nesting depth of program objects. Bytecode includes an instruction set with a one-byte opcode followed by an optional parameter. Intermediate representations such as bytecodes can be output by programming language implementations for ease of interpretation, or can be used to reduce hardware and operating system dependencies by allowing the same code to run across platforms on different devices. Bytecodes can often be directly executed on a virtual machine (p-code machine, i.e., interpreter), or they can be further compiled into machine code for better performance. In some cases, binary software code encoded using a platform-independent language such as JAVA can be stored in a bytecode format.

[0024] Although Figure 1The elements are shown as including various component parts, portions or modules that implement various features and functions, however, these elements may instead appropriately include several sub-modules, third-party services, components, libraries, etc. In addition, the features and functions of the various components may be appropriately combined into fewer components.

[0025] Figure 2 2 is a flowchart showing an example process 200 for processing labeled data according to an implementation. The process shown can be implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented. The example process 200 can be performed by Figure 1 , or any other system or module that processes the tagged data. For example, process 200 may be implemented by a server (e.g., Figure 1 The example process 200 may also be implemented using additional, fewer, or different operations, which may be performed in the order shown or in a different order.

[0026] The example process 200 starts at 210, where a set of labeled data is obtained. Each data in the set includes a feature vector and a label. The feature vector indicates feature information of the data, and the label indicates a classification result of the data.

[0027] In one example, the labeled data can be used to develop a machine learning model that detects whether software code is likely to be malicious. In some cases, the labeling can be binary, for example, the label for each data can be set to "0" - indicating that the code is benign, or can be set to "1" - indicating that the code can be potentially malicious. In some cases, the number of classes for labeling can be more than two. For example, the label can be set to 0, 1, 2, 3, indicating different degrees of risk that the code is malicious.

[0028] For the software code detection example, the feature vector of data may include information representing different features of the software code. Examples of features include file size, file format, file type (e.g., binary or source code), associated operating system or programming environment, the number of specific strings included in the code, the structure of the code (e.g., the order of the instruction set), signature, checksum, or other information of the software code that can be used to determine whether the software code is malicious. In some cases, the feature information of the software code discussed above can be obtained from metadata associated with the code, from a preprocessing operation that scans the software code and automatically extracts feature information from the code, or from both. The feature information can be encoded into a vector and flattened into a single feature vector.

[0029] At 220, each labeled data is processed by a plurality of machine learning models to obtain a plurality of classification results. Each machine learning model has been trained to perform classification on the data.

[0030] In some cases, multiple machine learning models may be of the same type but have different parameters. For example, each of the multiple machine learning models may be a neural network, but they may have different numbers of layers or neurons. In some cases, multiple machine learning models may have the same number of layers or neurons, but are developed using different labeled data sets in the training process. In some cases, multiple machine learning models may include different types of machine learning models, such as support vector machines and neural networks. Alternatively or additionally, multiple machine learning models may include different types of neural networks, such as convolutional neural networks (CNNs), feedforward neural networks (FNNs), recurrent neural networks (RNNs), etc.

[0031] In some cases, multiple machine learning models can be generated by using a virtual ensemble. A virtual ensemble can be formed based on a basic machine learning model. Random parameters can be introduced to generate additional machine learning models from a machine learning model. In one example, the basic machine learning model is a neural network including multiple layers, and each layer includes multiple neurons. For each neuron in the neural network, a random function can be used to determine whether to turn the neuron on or off. The random function can randomly return one of two results, "on" or "off". A default probability can be set. For example, if the default probability is set to 70%, the random function will return "on" with a probability of 70% and return "off" with a probability of 30%. During each operation, a random function is executed for each neuron. In the case where the random function returns "on", the neuron is used to process data. If the random function returns "off", the neuron is turned off and bypassed.

[0032] A virtual ensemble may include multiple such neural networks. These neural networks have the same structure, the same number of layers, the same number of neurons, and the same parameters for each neuron. However, due to the random function, different neurons will be turned on or off for each neural network during each operation, and thus these neural networks will effectively execute as different neural networks and may return different results. Using a virtual ensemble reduces the complexity of training and developing different machine learning models.

[0033] During operation, for each labeled data, each machine learning model processes the same feature vector of the labeled data to generate a classification result for the labeled data. Because these machine learning models are different, the classification results for the same labeled data can be different.

[0034] At 230, a labeling uncertainty score is determined for each labeled data based on the classification results obtained for the labeled data at 230. The labeling uncertainty score is determined based on a difference between the average entropy score and the adjusted score.

[0035] The average entropy score can be calculated as follows:

[0036] In classification operations, Equation 1 can be used to calculate the entropy of the model:

[0037]

[0038] Where x represents the input feature vector, y represents the output class, and D represents the data set that has been used to train the respective model. K represents the number of models, and k (k=1,…,K) represents each model in the K models. The prediction for model k is denoted as H k (y|x, D) represents the entropy for model k.

[0039] Equation 2 can be used to calculate the average entropy of K models:

[0040]

[0041] The adjustment score indicates the degree of deviation between the classification results produced by each model. In some cases, the degree of deviation can be indicated by using the standard deviation calculation of the classification results of each model. When classifying the same data, a higher adjustment score reflects a higher degree of inconsistency among the models. In some cases, a scaling factor can be applied to the adjustment score.

[0042] Equation 3 can be used to calculate the labeling uncertainty score:

[0043]

[0044] in represents the mean entropy determined at (2), std represents the standard deviation function, represents the standard deviation of the classification result, and s represents the scaling factor. In some implementations, the scaling factor can be set to a value from 0.001 to 0.3.

[0045] In some cases, the scaling factor is determined based on the problem context, algorithm, and data. The scaling factor is typically less than 1. In one example, a test range from 0.001 to 0.999 can be tested. Several candidate values ​​within the test range can be selected using a logarithmic scale, for example, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, etc. These candidate values ​​can be used in experiments to evaluate performance based on performance metrics. In some cases, the value of the actual labeling error rate of the prioritized samples can be used as a performance metric. The higher the performance metric, the more accurate the formula with this specific scaling factor value is when selecting sampling with respect to labeling noise.

[0046] A higher labeling uncertainty score indicates higher data uncertainty because it reflects a higher agreement among the models, with each model expressing a lower confidence.

[0047] At 240, the labeling uncertainty score is used to determine whether to relabel one or more labeled data in the labeled data set. In some cases, for each labeled data, a labeling uncertainty score is calculated based on the classification results produced by different models, as previously discussed, by using equations (1)-(3). The threshold can be configured. If the labeling uncertainty score for a particular labeled data exceeds the threshold, the particular labeled data can be determined to have a higher data uncertainty. Alternatively or additionally, the labeled data can be graded based on its labeling uncertainty score. A configured percentage of labeled data with the highest labeling uncertainty score (e.g., 1%) is determined to have a higher data uncertainty and will be relabeled.

[0048] In some implementations, a notification may be generated after processing the set of labeled data. The notification may be output at a user interface of the server, sent to a different device, or a combination thereof. The notification may indicate labeled data having a labeling uncertainty score that exceeds a configured threshold. The notification may also include labeling uncertainty scores for the labeled data.

[0049] In some cases, these labeled data with higher data uncertainty are processed by a re-labeling operation to determine whether the labeling of the labeled data is correct. The re-labeling operation can include an automatic labeling operation to regenerate the labeling, submitting to a domain expert for further review, or both. For the software code detection example, the re-labeling operation can include submitting the software code corresponding to the labeled data to a server for virus checking, executing the software code in a controlled environment to determine whether it is malicious, or other measures.

[0050] Although software code detection is used as an example to illustrate the operation of label uncertainty determination for labeled data, the processes discussed in this disclosure can be used for other machine learning applications that use labeled data for classification, such as image recognition, speech or audio recognition, language translation, or other applications.

[0051] Figure 3 302 is a high-level architectural block diagram showing a computer 302 coupled to a network 350 according to an implementation. The illustration is only one possible implementation of the subject matter and is not intended to limit the present disclosure to a single implementation. Those of ordinary skill in the art will appreciate that the components consistent with the present disclosure may be connected, combined or used in alternative ways.

[0052] The network 350 facilitates communication between the computer 302 and other devices. In some cases, a user (e.g., an administrator) can access the computer 302 from a remote network. In these or other cases, the network 350 can be a wireless or wired network. In some cases, a user can access the computer 302 locally. In these or other cases, the network 350 can also be a memory conduit, a hardware connection, or any internal or external communication path between components.

[0053] Computer 302 includes a computing system configured to execute the algorithms described in this disclosure. For example, computer 302 can be used to implement Figure 1 In some cases, the algorithm may be implemented in an executable computing code (e.g., C / C++ executable code). Alternatively or in combination, the algorithm may be implemented in an application (e.g., EXCEL). In some cases, the computer 302 may include an independent LINUX system running a batch processing application. In some cases, the computer 302 may include a mobile or personal computer running an application.

[0054] The computer 302 may include input devices, such as a keypad, a keyboard, a touch screen, a microphone, a voice recognition device, or another device that can accept user information and / or output devices that transmit information associated with the operation of the computer 302, including digital data, visual and / or audio information, or a GUI.

[0055] The computer 302 may function as a client, a network component, a server, a database, or other persistence, etc. In some implementations, one or more components of the computer 302 may be configured to operate within a cloud computing-based environment.

[0056] At a high level, the computer 302 is an electronic computing device operable to receive, transmit, process, store or manage data and information. According to some implementations, the computer 302 may also include an application server, an email server, a web server, a cache server, a streaming data server, a business intelligence (BI) server, and / or other servers, or be communicatively coupled thereto.

[0057] Computer 302 may receive requests from client applications (e.g., executed on a user device) over network 350 and respond to received requests by processing the requests in an appropriate software application. In addition, requests may also be sent to computer 302 from internal users (e.g., from a command console or by another appropriate access method), external or third parties, other automated applications, and any other appropriate entity, person, system, or computer.

[0058] Each of the components of computer 302 can communicate using system bus 303. In some implementations, any and / or all components (both hardware and / or software) of computer 302 can interface with each other and / or interface 304 through system bus 303 using application programming interface (API) 312 and / or service layer 313. API 312 can include specifications for routines, data structures, and object classes. API 312 can be computer language independent or dependent, and refers to a complete interface, a single function, or even an API set. Service layer 313 provides software services to computer 302. The functions of computer 302 can be accessible to all service consumers using the service layer. Software services such as those provided by service layer 313 provide reusable defined business functions through defined interfaces. For example, the interface can be software written in JAVA, C++, or other appropriate languages ​​that provide data in extensible markup language (XML) format or another appropriate format. Although shown as an integrated component of the computer 302, alternative implementations may show the API 312 and / or the service layer 313 as separate components relative to the other components of the computer 302. Furthermore, any or all portions of the API 312 and / or the service layer 313 may be implemented as a child module or sub-module of another software module or hardware module without departing from the scope of the present disclosure.

[0059] Computer 302 includes interface 304. Although Figure 3304, but two or more interfaces 304 may be used depending on the specific needs, configuration, or specific implementation of the computer 302. The interface 304 is used by the computer 302 to communicate with other systems in a distributed environment connected to a network 350 (whether or not shown). Generally, the interface 304 includes logic encoded in software and / or hardware in a suitable combination and operable to communicate with the network 350. More specifically, the interface 304 may include software supporting one or more communication protocols associated with the communication, such that the network 350 or the hardware of the interface is operable to communicate physical signals.

[0060] Computer 302 includes processor 305. Although Figure 3 A single processor 305 is shown in FIG. 3 , but two or more processors may be used depending on the specific needs, configuration, or specific implementation of the computer 302. In general, the processor 305 executes instructions and manipulates data to perform operations of the computer 302. In some cases, the processor 305 may include a data processing device.

[0061] The computer 302 also includes a memory 306 for storing data specific to the computer 302. Figure 3 302, two or more memories may be used depending on the specific needs, configuration, or specific implementation of the computer 302. Although the memory 306 is shown as an integral component of the computer 302, in alternative implementations, the memory 306 may be external to the computer 302.

[0062] The application 307 includes an algorithmic software engine that provides functionality according to the specific needs, configuration, or specific implementation of the computer 302. Although shown as a single application 307, the application 307 may be implemented as multiple applications 307 on the computer 302. Furthermore, although shown as integrated into the computer 302, in alternative implementations, the application 307 may be external to the computer 302.

[0063] There may be any number of computers 302 associated with the system 300 or external to the system 300 and communicating over the network 350. Furthermore, the terms "client," "user," and other appropriate terms may be used interchangeably as appropriate without departing from the scope of the present disclosure. Furthermore, the present disclosure contemplates that many users may use one computer 302, or that one user may use multiple computers 302.

[0064] The described implementations of the subject matter may include one or more features alone or in combination.

[0065] For example, in a first implementation, a method includes: obtaining, by an electronic device, a set of labeled data, wherein each labeled data in the labeled data includes a feature vector and a label; for each labeled data in the set of labeled data: processing the labeled data by using multiple machine learning models to obtain multiple classification results, wherein each classification result in the multiple classification results is obtained by processing the feature vector of the labeled data by using different machine learning models among the multiple machine learning models; and determining a label uncertainty score of the labeled data based on a difference between an average entropy score and an adjusted score; and determining whether to re-label one or more labeled data in the set of labeled data based on the label uncertainty score.

[0066] The above and other described implementations may each optionally include one or more of the following features:

[0067] A first feature, which is combinable with any of the following features, wherein the adjustment score is determined based on a standard deviation of classification results of labeled data.

[0068] A second feature that is combinable with any of the previous or following features, wherein the adjustment score is further determined based on the scaling factor.

[0069] A third feature, which is combinable with any of the previous or following features, wherein each of the tag data represents a software code, and the tag indicates whether the software code is potentially malicious.

[0070] The fourth feature, which is combinable with any of the previous or following features, is where multiple machine learning models are part of a virtual ensemble.

[0071] A fifth feature, which is combinable with any of the previous or following features, wherein each neuron in the plurality of machine learning models is associated with a random function that returns an indicator to indicate whether the neuron is on or off.

[0072] A sixth feature, which is combinable with any of the previous features, wherein determining whether to re-label one or more labeled data comprises: comparing a labeling uncertainty score of the one or more labeled data with a configured threshold.

[0073] In a second implementation, a computer-readable medium containing instructions, which when executed, causes a computing device to perform operations, including: obtaining labeled data, wherein each labeled data in the labeled data includes a feature vector and a label; for each labeled data in a labeled data set: processing the labeled data using multiple machine learning models to obtain multiple classification results, wherein each classification result in the multiple classification results is obtained by processing the feature vector of the labeled data using different machine learning models in the multiple machine learning models to process the feature vector of the labeled data; and determining a labeling uncertainty score of the labeled data based on a difference between an average entropy score and an adjusted score; and determining whether to re-label one or more labeled data in the labeled data set based on the labeling uncertainty score.

[0074] The foregoing and other described implementations may each optionally include one or more of the following features:

[0075] A first feature, which is combinable with any of the following features, wherein the adjustment score is determined based on a standard deviation of classification results of labeled data.

[0076] A second feature that is combinable with any of the previous or following features, wherein the adjustment score is further determined based on the scaling factor.

[0077] A third feature, which is combinable with any of the previous or following features, wherein each of the tag data represents a software code, and the tag indicates whether the software code is potentially malicious.

[0078] The fourth feature, which is combinable with any of the previous or following features, is where multiple machine learning models are part of a virtual ensemble.

[0079] A fifth feature, which is combinable with any of the previous or following features, wherein each neuron in the plurality of machine learning models is associated with a random function that returns an indicator to indicate whether the neuron is on or off.

[0080] A sixth feature, which is combinable with any of the previous features, wherein determining whether to re-label one or more labeled data comprises: comparing a labeling uncertainty score of the one or more labeled data with a configured threshold.

[0081] In a third implementation, an electronic device includes: one or more computers; and one or more computer memory devices interoperably coupled to the one or more computers and having a tangible, non-transitory machine-readable medium storing one or more instructions, which, when executed by the one or more computers, perform one or more operations, including: obtaining a set of labeled data, wherein each labeled data in the labeled data includes a feature vector and a label; for each labeled data in the labeled data set: processing the labeled data using multiple machine learning models to obtain multiple classification results, wherein each classification result in the multiple classification results is obtained by processing the feature vector of the labeled data using different machine learning models in the multiple machine learning models to process the feature vector of the labeled data; and determining a labeling uncertainty score of the labeled data based on a difference between an average entropy score and an adjusted score; and determining whether to re-label one or more labeled data in the labeled data set based on the labeling uncertainty score.

[0082] The foregoing and other described implementations may each optionally include one or more of the following features:

[0083] A first feature, which is combinable with any of the following features, wherein the adjustment score is determined based on a standard deviation of classification results of labeled data.

[0084] A second feature that is combinable with any of the previous or following features, wherein the adjustment score is further determined based on the scaling factor.

[0085] A third feature, which is combinable with any of the previous or following features, wherein each of the tag data represents a software code, and the tag indicates whether the software code is potentially malicious.

[0086] The fourth feature, which is combinable with any of the previous or following features, is where multiple machine learning models are part of a virtual ensemble.

[0087] A fifth feature, which is combinable with any of the previous or following features, wherein each neuron in the plurality of machine learning models is associated with a random function that returns an indicator to indicate whether the neuron is on or off.

[0088] A sixth feature, which is combinable with any of the previous features, wherein determining whether to re-label one or more labeled data comprises: comparing a labeling uncertainty score of the one or more labeled data with a configured threshold.

[0089] The implementation of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The implementation of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory computer storage medium for execution by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for execution by a data processing device for transmission to an appropriate receiver device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them.

[0090] The terms "data processing apparatus", "computer" or "electronic computer equipment" (or equivalents as understood by one of ordinary skill in the art) refer to data processing hardware, and include all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or also include a dedicated logic circuit device, such as a central processing unit (CPU), an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In some implementations, the data processing apparatus and / or the dedicated logic circuit device may be hardware-based and / or software-based. The apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. The present disclosure contemplates the use of a data processing apparatus with or without a conventional operating system, such as LINUX, UNIX, WINDOWS, MACOS, ANDROID, IOS, or any other suitable conventional operating system.

[0091] A computer program, which may also be referred to or described as a program, software, software application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple collaborative files, for example, a file storing one or more modules, subroutines, or code portions. A computer program may be deployed to be executed on a computer or multiple computers located at a site or distributed across multiple sites and interconnected by a communication network. Although the various parts of the program shown in the various figures are shown as separate modules that implement various features and functions through various objects, methods, or other processes, the program may appropriately include several submodules, third-party services, components, libraries, etc. On the contrary, the features and functions of various components may be appropriately combined into a single component.

[0092] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by special-purpose logic circuits (e.g., CPUs, FPGAs, or ASICs), and the apparatus can also be implemented as special-purpose logic circuits.

[0093] A computer suitable for the execution of a computer program can be based on a general or special microprocessor, both, or any other kind of CPU. Typically, the CPU will receive instructions and data from a ROM or RAM or both. The basic elements of a computer are a CPU for executing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operably coupled to, one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or receive data therefrom, or transmit data thereto, or both. However, a computer need not have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a PDA, a mobile audio or video player, a game console, a GPS receiver, or a portable storage device, such as a USB flash drive), to name just a few examples.

[0094] Computer-readable media (transient or non-transient, as the case may be) suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM, DVD+ / -R, DVD-RAM, and DVD-ROM disks. The memory may store a variety of objects or data, including caches, classes, frameworks, applications, backup data, jobs, web pages, web page templates, database tables, repositories storing business and / or dynamic information, and any other appropriate information, including any parameters, variables, algorithms, instructions, rules, constraints, or references thereto. Additionally, the memory may include any other appropriate data, such as logs, policies, security or access data, report files, and others. The processor and memory may be supplemented by or incorporated into a special-purpose logic circuit device.

[0095] To provide interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD, LED, or plasma monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse, trackball, or trackpad) by which the user can provide input to the computer. Input may also be provided to the computer using a touch screen, such as a tablet computer surface with pressure sensitivity, a multi-touch screen using capacitance or inductance, or other types of touch screens. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0096] The term "graphical user interface" or "GUI" may be used in the singular or plural to describe one or more graphical user interfaces and each of the displays of a particular graphical user interface. Thus, a GUI may represent any graphical user interface, including but not limited to a web browser, a touch screen, or a command line interface (CLI) that processes information and effectively presents information results to a user. Typically, a GUI may include a plurality of user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, drop-down lists, and buttons operable by a business suite user. These and other UI elements may relate to or represent the functionality of a web browser.

[0097] Implementations of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., such as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of wired and / or wireless digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a radio access network (RAN), a metropolitan area network (MAN), a wide area network (WAN), Worldwide Interoperability for Microwave Access (WIMAX), a WLAN using, for example, 802.11a / b / g / n and / or 802.20, all or a portion of the Internet, and / or any other communication system or systems at one or more locations. The network may communicate, for example, Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, voice, video, data, and / or other suitable information between network addresses.

[0098] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship with each other.

[0099] In some implementations, any or all of the components (hardware and / or software) of the computing system can use an API and / or service layer to interface with each other and / or interfaces. The API may include specifications for routines, data structures, and object classes. The API may be computer language independent or related and refers to a complete interface, a single function, or even an API set. The service layer provides software services to the computing system. The functions of the various components of the computing system can be accessed via the service layer and are accessible to all service consumers. Software services provide reusable defined business functions by defining interfaces. For example, the interface may be software written in JAVA, C++, or other appropriate languages ​​that provide data in XML format or other appropriate formats. The API and / or service layer may be an integrated and / or independent component relative to other components of the computing system. In addition, any or all parts of the service layer may be implemented as child modules or submodules of another software module, enterprise application, or hardware module without departing from the scope of this disclosure.

[0100] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope that may be claimed, but rather as descriptions of features that may be specific to a particular implementation of a particular invention. Certain features described in this specification in the context of separate implementations may also be implemented in combinations in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, in some cases, one or more features from a claimed combination may be deleted from the combination, and a claimed combination may point to a sub-combination or a variant of a sub-combination.

[0101] Specific implementations of the subject matter have been described. Other implementations, alternatives, and permutations of the implementations are within the scope of the appended claims, which will be apparent to those skilled in the art. Although operations are described in a particular order in the drawings or claims, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the operations shown (some operations may be considered optional) be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0102] In addition, the separation and / or integration of various system modules and components in the above-described implementations should not be understood as requiring such separation and / or integration in all implementations, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0103] Therefore, the above description of example embodiments does not define or constrain the present disclosure. Other changes, substitutions and alterations are also possible without departing from the spirit and scope of the present disclosure.

Claims

1. A method comprising: Acquiring, by an electronic device, a set of labeled data, wherein each of the labeled data includes a feature vector and a label; For each labeled data in the set of labeled data: Processing the labeled data by using a plurality of machine learning models to obtain a plurality of classification results, wherein each of the plurality of classification results is obtained by processing the feature vector of the labeled data by using a different machine learning model among the plurality of machine learning models; as well as determining a labeling uncertainty score for the labeled data based on a difference between the average entropy score and the adjusted score; as well as Based on the labeling uncertainty score, it is determined whether to re-label one or more labeled data in the labeled data set. 2 . The method of claim 1 , wherein the adjusted score is determined based on a standard deviation of the classification results of the labeled data. The method of claim 2 , wherein the adjustment score is further determined based on a scaling factor. 4 . The method of claim 1 , wherein each of the tag data represents a software code, and the tag indicates whether the software code is potentially malicious.

5. The method of claim 1, wherein the plurality of machine learning models are part of a virtual ensemble.

6. The method of claim 5, wherein each neuron in the plurality of machine learning models is associated with a random function that returns an indicator to indicate whether the neuron is on or off.

7. The method of claim 1 , wherein determining whether to re-label one or more labeled data comprises: The labeling uncertainty score of the one or more labeling data is compared to a configured threshold.

8. A computer-readable medium comprising instructions that, when executed, cause a computing device to perform operations comprising: Acquire a set of labeled data, wherein each labeled data in the labeled data includes a feature vector and a label; For each labeled data in the set of labeled data: Processing the labeled data by using a plurality of machine learning models to obtain a plurality of classification results, wherein each of the plurality of classification results is obtained by processing the feature vector of the labeled data by using a different machine learning model among the plurality of machine learning models; as well as determining a labeling uncertainty score for the labeled data based on a difference between the average entropy score and the adjusted score; as well as Based on the labeling uncertainty score, it is determined whether to re-label one or more labeled data in the labeled data set.

9. The computer-readable medium of claim 8, wherein the adjusted score is determined based on a standard deviation of the classification results of the labeled data.

10. The computer-readable medium of claim 9, wherein the adjustment score is further determined based on a scaling factor.

11. The computer-readable medium of claim 8, wherein each of the tag data represents a software code, and the tag indicates whether the software code is potentially malicious.

12. The computer-readable medium of claim 8, wherein the plurality of machine learning models are part of a virtual ensemble.

13. The computer-readable medium of claim 12, wherein each neuron in the plurality of machine learning models is associated with a random function that returns an indicator to indicate whether the neuron is on or off.

14. The computer-readable medium of claim 8, wherein determining whether to re-mark one or more marked data comprises: The labeling uncertainty score of the one or more labeling data is compared to a configured threshold.

15. An electronic device, comprising: one or more computers; as well as One or more computer memory devices interoperably coupled with the one or more computers and having a tangible, non-transitory, machine-readable medium storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising: Acquire a set of labeled data, wherein each labeled data in the labeled data includes a feature vector and a label; For each labeled data in the set of labeled data: Processing the labeled data by using a plurality of machine learning models to obtain a plurality of classification results, wherein each of the plurality of classification results is obtained by processing the feature vector of the labeled data by using a different machine learning model among the plurality of machine learning models; and determining a labeling uncertainty score for the labeled data based on a difference between the average entropy score and the adjusted score; and Based on the labeling uncertainty score, it is determined whether to re-label one or more labeled data in the labeled data set. 16 . The electronic device of claim 15 , wherein the adjustment score is determined based on a standard deviation of the classification result of the labeled data.

17. The electronic device of claim 16, wherein the adjustment score is further determined based on a scaling factor.

18. The electronic device of claim 16, wherein each of the tag data represents a software code, and the tag indicates whether the software code is potentially malicious.

19. The electronic device of claim 16, wherein the plurality of machine learning models are part of a virtual integration.

20. The electronic device of claim 19, wherein each neuron in the plurality of machine learning models is associated with a random function that returns an indicator to indicate whether the neuron is on or off.

Images