Tag modification techniques

Abstract

Embodiments of the present disclosure relate to label modification techniques. Systems, methods, and software can be used to modify labels associated with files, where a label is information that helps to classify a file. In some aspects, a method includes obtaining features from files, clustering the files based on similarity measures between the features, the similarity measures being independent of labels associated with the files, identifying mismatches between labels associated with files belonging to a given cluster, and modifying some labels of files in the given cluster based on a label distribution in the given cluster.

Description

Technical Field

[0001] This disclosure relates to modifications to the tags associated with files in datasets, data lakes, and other data storage systems, such as data warehouses, databases, etc. Background Technology

[0002] The quality of labels associated with data is a crucial parameter for training machine learning models. In fact, problems with labels can lead to reduced accuracy, misclassification, or biased predictions. Several common label-related problems can exist in machine learning model training. For example, incorrect labels may be associated with data due to human error or inaccurate automated labeling. Incorrect labels directly mislead the learning process because the machine learning model learns associations based on erroneous information. Therefore, it is necessary to seek ways to improve the quality of these labels. In the context of cybersecurity, machine learning models can be trained to classify files as either malware or non-malware. Incorrect labeling can have devastating consequences during the training of a machine learning model using a training dataset that includes files with labels such as those indicating whether a file is malware or non-malware. In fact, once a trained malware classification model is deployed (which has been trained on a flawed foundation), when a given file is evaluated by that trained model (before it is executed on a device), the trained malware classification model may improperly classify the file, allowing it to be executed. Therefore, incorrect labeling can increase false positives (causing warning fatigue and affecting other performance metrics of the trained model, such as accuracy); it can also increase false negatives (putting devices at risk of executing binary programs that are considered non-malicious when they are), thus reducing recall. Furthermore, this can affect threshold tuning, where incorrectly labeled records can maliciously and benignly shift the target decision boundary and cause the model to not perform as expected upon deployment. Therefore, mitigation of the problems associated with incorrect labeling is necessary. Attached Figure Description

[0003] Figure 1 A schematic diagram is depicted illustrating an example system that provides malware classification technology according to one implementation method.

[0004] Figure 2 This is a flowchart illustrating an example method for modifying labels associated with data, based on one implementation.

[0005] Figure 3 This is a flowchart illustrating an example method for generating file embedding vectors from binary files.

[0006] Figure 4This is a flowchart illustrating an example method for training a machine learning model based on defective data in the labels.

[0007] Figure 5 The diagram illustrates a high-level architecture block diagram of a computer based on one implementation.

[0008] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0009] Figure 1 A schematic diagram illustrating an example system providing malware classification technology according to one implementation is depicted. More specifically, system 100 includes a software service platform 106 communicatively coupled to a client device 102 via a network 110. The client device 102 represents an electronic device providing a file to be analyzed. In some cases, the client device 102 may send a file to the software service platform 106 for malware detection and analysis. In other cases, the software service platform 106 may send the output of the malware detection and analysis to the client device 102.

[0010] Software service platform 106 represents an application, application set, software, software module, hardware, or any combination thereof for detecting malware files. Software service platform 106 can be an application server, service provider, or any other network entity. Software service platform 106 can be implemented using one or more computers, computer servers, or cloud computing platforms. Software service platform 106 can be used to run trained machine learning models used in malware detection processes or malware analysis. In one variant, software service platform 106 can also perform... Figure 4 The training process is discussed in the relevant description. The software service platform 106 includes a software analyzer 104. The software analyzer 104 represents an application, set of applications, software, software module, hardware, or any combination thereof that performs data preprocessing on received files. In some implementations, the software analyzer 104 can generate file embedding vectors for the files, which are used as input to a trained machine learning model. Figures 2-4 The accompanying description provides additional details on these implementations. In one variant, both the software analyzer 104 and the software service platform 106 execute within the client device 102 itself. Indeed, due to technological advancements, an increasing number of client devices are capable of running trained machine learning models locally. For example, iPhones, which can be considered client devices, are well-suited for running machine learning models locally because they provide the core machine learning framework—specifically, dedicated chip components optimized for performing machine learning tasks, such as the Apple Neural Engine (ANE).

[0011] Turning to the general description, client device 102 may include, but is not limited to, any of the following: endpoint, computing device, mobile device, mobile electronic device, user equipment, mobile station, subscriber station, portable electronic device, mobile communication device, wireless modem, wireless terminal, or other electronic device. Examples of endpoints may include mobile devices, IoT (Internet of Things) devices, EoT (Internet of Things) devices, cellular phones, personal data assistants (PDAs), smartphones, laptops, tablets, personal computers (PCs), pagers, portable computers, portable gaming devices, wearable electronic devices, health / medical / fitness devices, cameras, vehicles, or other mobile communication devices having components for transmitting voice or data via a wireless communication network. Means of transport can include motor vehicles (e.g., cars, sedans, trucks, buses, motorcycles, etc.), aircraft (e.g., airplanes, unmanned aerial vehicles, unmanned aerial vehicle systems, drones, helicopters, etc.), spacecraft (e.g., space shuttles, space shuttles, spacecraft modules, space stations, satellites, etc.), watercraft (e.g., ships, vessels, hovercraft, submarines, etc.), rail transport (e.g., trains, trams, etc.), and other types of transport including any combination of the foregoing, whether currently existing or emerging. Wireless communication networks can include wireless links on at least one of licensed and unlicensed spectrum. The term "mobile device" can also refer to any hardware or software component capable of terminating a user's communication session. Furthermore, the terms "user equipment," "UE," "user equipment device," "user agent," "UA," "user equipment," and "mobile device" are used interchangeably herein.

[0012] Example system 100 includes network 110. Network 110 represents an application, application set, software, software module, hardware, or combination thereof that can be configured to transmit data messages between entities in example system 100. Network 110 may include a wireless network, a wired network, the Internet, or a combination thereof. For example, network 110 may include one or more radio access networks (RANs), core networks (CNs), and the Internet. The RAN may include one or more radio access technologies. In some implementations, the radio access technology may be Global System for Mobile Communications (GSM), Provisional Standard 95 (IS-95), Universal Mobile Telecommunications System (UMTS), CDMA2000 (Code Division Multiple Access), Evolved Universal Mobile Telecommunications System (E-UMTS), Long Term Evolution (LTE), Advanced LTE, 5G, or any other radio access technology. In some cases, the core network may be an Evolved Packet Core (EPC).

[0013] Although Figure 1The elements are shown as various components, parts, or modules that implement various features and functions; however, these elements can be appropriately modified to include multiple submodules, third-party services, components, libraries, etc. Furthermore, the features and functions of various components can be appropriately combined into fewer components.

[0014] Figure 2 This is a flowchart illustrating an example operation 200 for modifying tags associated with a file stored in a database, according to one implementation. Example operation 200 can be provided by a software service platform (e.g., Figure 1 The software service platform 106 shown is implemented. Figure 2 The example operation 200 shown can be implemented using additional, fewer, or different operations (which can be performed in the order shown or in a different order).

[0015] like Figure 2 As shown, the file set 202 is obtained by, or can be accessed by, the software service platform 106. In one embodiment of this disclosure, in obtaining step 210, each file is processed by a file encoder to generate a file embedding vector. Therefore, for each file, at least one vector is generated. Thus, the vector set is obtained from the use of the file encoder. There are many ways to obtain features from files to generate vectors.

[0016] In one implementation, the file set includes only source code files. Furthermore, in another implementation, all files from that file set have tags indicating whether a file is malware / malicious. For example, the tag can be a bit set to 0 or 1. The value used to define whether a file is malware depends on the implementation chosen. In a variant, the tag can take many values. For example, the tag value can be associated with a malware category or series. Files belonging to the same malware category or series then have the same tag value. For example, different values / tags can be associated with each of the following file types or categories or series: ransomware files, Trojan files, spyware and adware files, worm files, backdoor files, botnet files, privilege escalation package files, and / or information stealer files. In a variant, the granularity of the tags can be narrower. For example, the value / tag can correspond to specific known malware, such as Mirai (which also belongs to the botnet series) or Atom Stealer / AMOS related files. When the file set includes only source code files, encoding of these files can be performed before executing step 210. For example, a compiler can be used to generate binary files from a set of source code files. For instance, when the source code files are written in an interpreted language (e.g., Python, JavaScript), the encoder compiles the source code files to obtain bytecode files as a binary file. When the source code files are written in a compiled language (e.g., C / C++, Java), the encoder compiles the source code files to obtain an executable binary file.

[0017] In one implementation, when the file is a binary file, obtaining set 210 can correspond to the situation based on... Figure 3 The computer implementation method of the operations described in the document.

[0018] In another variation, instead of converting the source code file into a device-executable binary file, step 210 directly generates vectors by processing the source code file. For example, in one variation, text / instructions from the source code file are encoded into numerical values ​​using ASCII or UTF 8-byte representations of characters. Thus, each source code file is converted into a vector, where each byte represents a character from the source code. In another variation, the source code file is parsed into an Abstract Syntax Tree (AST), which can then be serialized into binary. In yet another embodiment, each source code file is divided into terms or chunks / parts of code, and hash functions can be applied to these terms or chunks / parts. Thus, for a given file, the concatenation of all hashes defines a vector associated with that given file. In one variation, a pre-trained transformer model (such as CodeBERT) specifically designed to process source code for various Natural Language Processing (NLP) tasks, such as code search, code summarization, and code classification, can be used. In fact, according to this embodiment of the present disclosure, such a pre-trained transformer model can convert source code files into vectors. In another variation, a trained autoencoder is used to obtain vectors. More precisely, in this context, the vectors are the output of the encoder of the trained autoencoder (meaning the vectors are the representation of the file in the latent space). According to one implementation, the vectors obtained from the process described in this paragraph are the output of step 210. However, in another embodiment, based on... Figure 3 The computer implementation of the operation described herein is based on these vectors (in order to obtain the vector, which is then regarded as the result of obtaining step 210) as input (rather than in the process of obtaining the vector). Figure 3 The binary file described in the image is executed. In cases where the vector includes floating-point values, serialization can be performed to obtain a vector consisting only of bit sequences (which can be aggregated into bytes), thus conforming to... Figure 3 Description of the operations in the text.

[0019] It should be noted that the tags associated with the file are then associated with the corresponding vectors generated by obtaining step 210.

[0020] Vector set 212 will be used to process and verify the accuracy of the tags from file set 202.

[0021] Then, in step 214, clustering of the vectors (and thus clustering of the corresponding files) is performed based on the similarity measure (or distance) between the vectors from the vector set 212.

[0022] In one variant, a normalization step can be performed before clustering the vectors.

[0023] In one embodiment, clustering techniques such as K-means clustering can be used.

[0024] In one variant, clustering techniques such as Density-Based Noisy Spatial Clustering Application (DBSCAN) can be used. However, it should be noted that DBSCAN is generally better suited for clustering data when clusters have similar densities. Therefore, a selection step can be performed to choose which clustering technique to use among the possible ones. In one implementation, the selection step includes evaluating the resegmentation of files based on their labels. If high dissimilarity is identified between groups of files, DBSCAN is not selected to perform the clustering step on the vector set 212.

[0025] In one variant, clustering techniques can be performed that rely on the use of a hierarchical navigable small-world (HNSW) graph and the execution of an approximate nearest neighbor (ANN) search. In fact, a fast ANN search can be performed by building an index that efficiently organizes the data from the vector set.

[0026] In one implementation, a dimensionality reduction technique such as principal component analysis (PCA) can be used to obtain a dimensionality-reduced vector set from the vector set 212.

[0027] In another implementation, other dimensionality reduction methods such as t-SNE (t-distributed random neighborhood embedding) or UMAP (uniform manifold approximation and projection) can be used to obtain a set of dimensionality-reduced vectors from the vector set 212.

[0028] Once the set of dimensionality-reduced vectors is obtained, the clustering techniques mentioned earlier can be applied to this set.

[0029] Once clustering step 214 has been performed, the vectors or reduced-dimensional vectors are clustered into clusters.

[0030] It should be noted that in one variant, file clustering can be performed without vectorizing the files. In fact, instead of performing steps 210 and 214 (using vectors associated with the files), a trained cross-encoder model (a type of machine learning model that jointly evaluates a pair of files to determine the relationship or score between them) can be used. In effect, file clusters can be determined based on the output of the trained cross-encoder model (which characterizes the similarity between files). Furthermore, the clusters already obtained in this way can be refined by using a string similarity metric combined with comparisons about a threshold. Then, step 216 can be performed to process the labels associated with the files.

[0031] For each cluster, identification step 216 is performed to compare the labels of its vector or reduced-dimensional vector, or the labels of the file. Step 216 may include statistics on the distribution / re-segmentation of labels within a given cluster. For example, a ranking of different labels (based on their frequency of occurrence in a given cluster) may be established. In a variant, a modification to the comparison threshold (and thus the cluster definition) may be performed based on the label distribution within the cluster. In this case, identification step 216 is repeated for the potential sub-clusters obtained after modifying the comparison threshold to obtain the expected label distribution within the sub-clusters. After performing identification step 216, several results and subsequent actions may occur regarding the labels.

[0032] In practice, in one implementation, if more than 90% of the label values ​​are detected in a vector or reduced-dimensional vector of a given cluster, modification step 218 is performed to change the labels of the remaining vectors or reduced-dimensional vectors. In fact, the labels are modified by assigning the same label to all vectors or reduced-dimensional vectors belonging to a given cluster. Label modification can be performed automatically or manually. For example, in one embodiment, a warning message on the user interface may require the user to confirm or reject the label modification. Therefore, since each vector or reduced-dimensional vector is associated with a file, the file's label is changed accordingly. As explained earlier, when clustering is performed without vectorizing the file, the same modification is applied directly to the file labels. In one implementation, the file labels and the vector / reduced-dimensional vector labels are the same.

[0033] In one variation, as mentioned earlier, records with the highest probability of labeling errors can be recursively identified by repeatedly dividing clusters with significant label impurities into subclusters. For example, if a given cluster has less than, say, 90% of its majority classes, subclusters of that cluster are created, and for any of those subclusters where the majority class percentage is less than 90%, this process is repeated until all subclusters have a majority class percentage greater than or equal to 90%, or less than the size that can be further divided. In this example, we cannot further divide into subclusters of size 10. Possible labeling errors for the minority classes in all remaining subclusters can then be investigated.

[0034] Figure 3 This is a flowchart illustrating an example method for generating file embedding vectors from binary files.

[0035] exist Figure 3The embodiments described herein process binary files. However, as explained above, in this process, pre-serialized vectors can also be used instead of binary files. In fact, even if a vector contains floating-point values, a binary representation of those values ​​can be obtained by using binary serialization techniques. In a sense, the vector is converted into a sequence of 0s and 1s, as a binary file (but without the executable properties inherent in binary files).

[0036] Generally, binary files (often also called binary programs or executables) are defined as files that contain data (usually in the form of binary code, i.e., sequences of 0s and 1s) that can be directly read by the hardware of a device or virtual machine, rather than in the form of ordinary human-readable text. Most binary files have a structured format or header that provides information about how to read the data. There are various types of binary files. For example, executable files (.exe, .bin, .dll) that contain compiled code that a computer can directly execute represent one type of binary file. Executable files can run independently (like programs) or install software by unpacking some necessary components and placing them in a specific directory on the device (like setup or installer files). Furthermore, binary files or executables require specific software or utilities to interpret them and are typically opened by applications designed to process them in a specific format. Another example of a binary file is a PE (Portable Executable) file or an ELF (Executable Linkable Format) file, which are file formats used to store executable code and other data for programs on Windows and Linux systems, respectively. They contain the information necessary for the operating system to load and execute the program. Therefore, binary files are a broad term encompassing any file that can be directly executed by a computer. For example, bytecode, which serves as intermediate code between source code and machine code, can also be considered a binary file. In fact, even bytecode that isn't directly executed by the device's hardware can be considered binary files, as it is typically executed by a runtime environment such as the Java Virtual Machine for Java bytecode or the Python interpreter for Python bytecode. Furthermore, it should be noted that scripts written in languages ​​like PowerShell can be compiled into binary executables. In fact, tools like PyInstaller or py2exe can convert Python scripts into executable binaries. Attackers sometimes convert PowerShell or Python scripts into binaries to bypass security tools that block scripts. Therefore, binary files in this context can be associated with many different high-level languages. Malicious scripts can be "packaged" into binary files using packers or obfuscators. Binaries may contain embedded scripts that are unpacked or decrypted at runtime. Attackers use binary representations to make it more difficult for signature-based antivirus programs to directly detect script content. Therefore, binary file analysis is an important aspect to consider when protecting electronic devices. In the preceding discussion, the examples of binary files were primarily computer-related. However, other electronic devices can use binary files. For example, the Android application package (apk) file used for app distribution and installation can be considered a binary file.In practice, an .apk file is a container that stores compiled code in DEX (Dalvik executable) format (used by the Android runtime to execute the app), along with other components such as resources (i.e., images, sounds, and layouts) and a manifest file describing the app's structure and permissions. Another type of binary file is the iOS App Store Package (IPA) file, which includes compiled app code (a binary format for the ARM architecture of iOS devices), resources (images, audio, user interface files), and an info.plist file (containing metadata about the app, such as version information, permissions, configuration parameters, etc.). This disclosure can be applied to such files.

[0037] More precisely, given a binary file, a number of elements consisting of N1 bytes are selected from the binary file according to the scanning process described below, where N1 is an integer greater than or equal to 1. There are several ways to perform this scanning and selection process 301.

[0038] According to one implementation, the scanning and selection process 301 depends on the structure of the binary file. In practice, since different binary file types have different formats and structures, the scanning and selection process 301 may include additional processes for detecting the file's nature to classify it as a PE file, an ELF file, or another file type. It then determines the order in which the scanning is performed. For example, according to one embodiment of the invention, if the binary file is an ELF file, the ELF header is scanned from the beginning to the end to obtain elements consisting of N1 bytes. If the ELF header size is not a multiple of N1, zero bytes may be added at the end of the scan of the ELF header to obtain N1 bytes of elements. In parallel with or after this process, the ELF data is scanned, and N1 bytes of elements are selected from the ELF data. In one embodiment, the scan begins at the beginning of the ELF data and continues until the end of the ELF data. In a variant, scanning and selection are performed based on different fragment types. For example, in one embodiment, the process begins with the fragment header table and then processes bytes from the .data, .bss, .rodata, .symtab, .strtab, .rel.data, .text, and .rel.text fragments in that order. In a variant, other fragment processing orders may be considered. Furthermore, in a variant, some uninteresting bytes may be discarded from the scan and selection steps. In practice, important information about the dangers of a file is unlikely to be included in fragments such as .comment or .note fragments from ELF data. These fragments contain optional comments, metadata, and notes / annotations, respectively.

[0039] A similar process can be performed on other binary file types (i.e., depending on the binary file structure caused by its type, with a specific byte scan order). For example, the components of an APK file can be scanned according to a specific order.

[0040] According to one implementation, a scanning and selection process 301 is performed without considering the structure of the binary file. In this variation, it allows the binary file to be processed quickly. Therefore, in this embodiment of the invention, the binary file is processed as a whole. For example, starting from the beginning of the binary file, N1 bytes are selected at the start of the process, and then, according to a sliding window with a value equal to N1, other bytes are selected in groups of N1 bytes each time. This process is repeated until all bytes in the binary file have been scanned. Similarly, if the size of the binary file is not a multiple of N1, zero bytes are added to obtain a final group of N1 bytes, or the number of bytes in the final byte group is less than N1 bytes, and these are used as is in subsequent processing. In one variation, the starting point of the scanning process is not the beginning of the binary file, but the end of the binary file. This means that bytes in the binary file are scanned and selected in the opposite direction to one of the previous embodiments. In other variations, the starting point of the scanning and selection process is defined as a given position in the binary file. In this case, bytes can be scanned towards the end of the binary file. Once the end of the file is reached, the remaining unscanned bytes are scanned starting from the beginning of the binary file (this scanning and selection process can be viewed as a cyclical process). In another variation, if the starting point of the scanning and selection process is defined as a given position in the binary file, bytes can be scanned in the opposite direction (i.e., towards the beginning of the binary file) compared to the previous embodiments. Again, once the beginning of the binary file is reached, the scanning and selection process continues from the end of the binary file. These examples are not exhaustive, and those skilled in the art can use other methods of scanning and selecting bytes in a binary file within the spirit of the described examples.

[0041] In one embodiment of this disclosure, N1 bytes are selected each time and provided as input to the hash function. The output of the hash function is located in a data vector that serves as a file embedding vector. Therefore, according to this embodiment, obtaining such a file embedding vector is performed instantaneously. In a variation, an intermediate memory buffer can be used to prepare the data to be hashed by the hash function. Both methods achieve the determination of the hash value sequence 302.

[0042] Different hash function types can be used; for example, non-cryptographic hash functions such as the Pearson hash function or the MurmurHash function, as chosen earlier, can handle N1 bytes of input. The output of the Pearson hash function is typically 8 bits (1 byte). Therefore, the output of the Pearson hash function is simply a number between 0 and 255. However, larger hash values ​​(16-bit, 32-bit, 64-bit, etc.) can be generated by running the Pearson algorithm multiple times under different initial conditions.

[0043] In addition, the most common versions of MurmurHash are MurmurHash2 and MurmurHash3. They can generate outputs of 32-bit, 64-bit, and even 128-bit sizes.

[0044] In another variation, two hash functions can be used to process the same input of N1 bytes. According to this embodiment, a truncation of the concatenated two hash values ​​can also be performed to limit the size of the hash value. Truncation is defined as the output result and is used to define / generate the hash sequence. For example, given an input of N1 bytes, a Pearson hash function that outputs a single byte as the result can be used. Then, a Pearson hash function with a different permutation table that also outputs a single byte as the result can be used on the same input of N1 bytes. Thus, for a given input of N1 bytes, two bytes (i.e., 16 bits) are obtained depending on the use of the two hash functions. However, truncation can be performed instead of using two bytes from the hash sequence. In fact, in one embodiment, the lower 12 bits of the 16 bits are retained. These 12 bits define the hash value. In a variation, another selection function can be used to extract multiple bits from the 16 bits. For example, the selection function can take the highest 12 bits from the 16 bits. In one variant, the selection function discards the two most significant bits and the two least significant bits to obtain 12 remaining bits. The selected bits correspond to the hash value associated with a given input of N1 bytes.

[0045] In one embodiment, the number N1 is an integer belonging to the range from 2 to 15.

[0046] In one variant, a preprocessing step can be performed before the scanning and selection steps. This preprocessing can remove zero bytes contained in the binary file.

[0047] In one implementation, several file embedding vectors can be generated for a given binary file. In this example, once the first file embedding vector has been generated, another file embedding vector can be generated by repeating the process using a value N2 different from N1. It should be noted that the generation of these vectors can be performed in parallel.

[0048] Therefore, in one variant, given a binary file (or a serialized vector as explained above), 2 to 10 file embedding vectors or vectors can be generated by using several different N1 values ​​and by repeating the described process.

[0049] According to one implementation, all these vectors can be concatenated and processed by clustering technique 214.

[0050] In one variation, the vector can be obtained as follows: once all hash values ​​have been obtained or determined, a histogram vector is determined. The histogram vector is a numerical representation of the histogram of hashes, which is a graphical representation of the hash distribution. Therefore, for each possible hash value defined by its position in the vector, the vector includes numbers corresponding to the frequency or count of hashes with that value. This vector is obtained in step 303.

[0051] In one variant, several histogram vectors can be obtained for a given binary file (by repeatedly scanning and selecting the given binary file using different N1 values).

[0052] Therefore, in Figure 3 The file embedding vector mentioned earlier can be a histogram vector as explained above, or a concatenated vector of hash value vectors as also explained above. In a variant, it can also be a concatenation of histogram vectors.

[0053] According to one embodiment of this disclosure, a normalization process can be performed on the vectors before they are used in clustering step 214. For example, min-max normalization (rescaling) can be performed.

[0054] In the previous example, the scan and extract 301 focused on bytes from the binary file. However, in a variant, the scan and extract steps can be performed on nibbles (i.e., groups of 4 bits) instead of bytes.

[0055] In one variation, a trained autoencoder can be used to add data from... Figure 3 The vector is obtained using the method described in [the document]. In fact, in one embodiment of this disclosure, an encoder model of a trained autoencoder is used to obtain a representation of a given document in the latent space. This latent space representation (i.e., a vector) of the given document is then compared with the vector obtained from [the document]. Figure 3 The vectors obtained by the method described are combined (e.g., via cascading). The vectors generated by this combination are then used in clustering step 214.

[0056] According to one implementation method Figure 4This is a flowchart illustrating an example method for training a machine learning model to be used as a malware classification model.

[0057] As is well known, machine learning models are trained using a process that involves feeding them large amounts of data and allowing them to learn patterns and relationships within that data. However, as mentioned earlier, incorrect labeling in the training data can significantly impact the performance and reliability of machine learning models.

[0058] In one implementation, a method for training a machine learning model based on a supervised approach is proposed. Below, the machine learning model is a malware classification model, but the operations and processes used to train this model can be applied to other machine learning model types.

[0059] According to one embodiment of this disclosure, the device obtains a file dataset 401 containing many different files. In one embodiment of this disclosure, several datasets may be obtained, and each file dataset may aggregate files of the same type (i.e., for example, a first dataset containing only ELF files, a second dataset containing only EXE files, a third dataset containing only Python scripts, etc.).

[0060] The training process includes and in Figure 2 The process described is similar to preprocessing step 402. Step 402 obtains the vector dataset from the file dataset. Step 402 may affect the original labels or information associated with the files. In fact, the labels can be based on... Figure 2 The relevant clustering and modification steps are used to make the changes. Therefore, the output of step 402 is a set of vectors with updated labels. In a sense, the quality of the information provided by the labels has been enhanced because incorrect labels have been corrected.

[0061] Once step 402 is completed, a data partitioning process can be performed to divide the given vector / file embedding vector dataset into a training set and a test set. The training set is used to train the model, while the test set is used to evaluate its performance.

[0062] According to one embodiment of this disclosure, different models can be selected for training. For example, a feedforward neural network (FNN), also known as a multilayer perceptron (MLP), can be used. In one variation, a convolutional neural network (CNN) can be selected for training. In another variation, a recurrent neural network (RNN) or a long short-term memory (LSTM) network can be selected. Furthermore, other architectures depending on the use of transformers, or hybrid approaches depending on the use of MLPs combined with autoencoders, can be chosen. The method for selecting the parameters and hyperparameters of each model is not described in this document. However, those skilled in the art will understand that modifications to these parameters and hyperparameters can be made based on the results of training these models to obtain better results. In fact, to determine these parameters and hyperparameters, comparisons of results must be performed. Factors such as the number of layers, the number of neurons per layer, the activation function, and the optimization algorithm have a significant impact on the behavior of the model. This is for fine-tuning purposes beyond the scope of this document.

[0063] In one variant, the cross encoder can be trained in a similar manner.

[0064] In another implementation, gradient boosting frameworks such as XGBoost, CatBoost, and LightGBM can be used to train machine learning models.

[0065] Once a model architecture is chosen, model training is performed by using a training dataset, a loss function that measures the difference between the model's predictions and the true values, and an optimization algorithm (e.g., gradient descent) to iteratively update the parameters (weights) to minimize the loss function. In practice, during model training, the internal parameters (weights and biases) are modified to minimize the difference between the model's predictions and the actual values ​​in the training data.

[0066] Model training 403 also includes an evaluation step, which assesses the trained model on a test dataset to evaluate its performance. Based on the results, model refinement (i.e., tuning of the model's hyperparameters) can be performed, or the training process can be stopped at this stage if the performance metrics meet the stopping criteria.

[0067] In one embodiment, once a trained machine learning model is obtained / generated, it can be deployed to the software service platform 106.

[0068] The training process and deployment of trained machine learning models can be repeated periodically based on parameters of the security policy, which define, for example, the time frame or frequency of training. In other cases, a security alert can be an event that triggers the initiation of new training for one or more models.

[0069] In one variant, several machine learning models are trained: a model for each file type. More specifically, in step 401, there is a file dataset for each file type. For example, according to one embodiment of this disclosure, a machine learning model specifically trained on a .EXE file dataset can be obtained. Similarly, according to another embodiment of this disclosure, a machine learning model specifically trained on an Apk file dataset can be obtained.

[0070] Figure 5 The diagram illustrates a high-level architecture block diagram of a computer 500 according to one implementation. The computer 500 can be implemented as a client device 102, a software service platform, or any combination thereof. The computer 500 can also be used to implement... Figures 1-4 The operations discussed herein. The illustrations described are merely one possible implementation of the described subject matter and are not intended to limit this disclosure to the single described implementation. Those skilled in the art will understand that the described components can be connected, combined, and / or used in alternative ways consistent with this disclosure.

[0071] In some cases, Figure 3 The steps can be implemented in executable computational code (e.g., C / C++ executable code). In some cases, computer 500 may include a standalone Linux system running batch applications. In other cases, computer 500 may include a mobile or personal computer.

[0072] Computer 500 may include a computer that includes input devices (such as keypad, keyboard, touch screen, microphone, voice identification device, other devices that can accept user information) and / or output devices (that convey information related to the operation of the computer, including digital data, visual and / or audio information, or GUI).

[0073] Computer 500 can act as a client, network component, server, database, or other entity, and / or any other component. In some implementations, one or more components of Computer 500 can be configured to operate within a cloud-based environment.

[0074] At a higher level, computer 500 is an electronic computing device operable to receive, transmit, process, store, or manage data. According to some embodiments, computer 500 may also include, or be communicatively coupled to, application servers, email servers, web servers, cache servers, streaming data servers, business intelligence (BI) servers, and / or other servers.

[0075] Computer 500 can collect data on network events or mobile application usage events from a web browser or client application (e.g., installed plugins) via network 110. Furthermore, data can be collected by computer 500 from internal users (e.g., from a command console or through another suitable access method), external or third parties, other automated applications, and any other suitable entity, individual, system, or computer.

[0076] Each component of computer 500 can communicate using system bus 512. In some implementations, any and / or all components (hardware and / or software) of computer 500 can connect to each other and / or to interface 502 via system bus 512 using application programming interfaces (APIs) 508 and / or service layer 510. API 508 may include specifications for routines, data structures, and object classes. API 508 may be language-independent or language-dependent, and refers to a complete interface, a single function, or even a set of APIs. Service layer 510 provides software services to computer 500. The functionality of computer 500 may be accessible to all service consumers using the service layer. Software services (such as those provided by service layer 510) provide reusable, defined business functionality through defined interfaces. For example, the interface may be software written in JAVA, C++, or other suitable languages ​​that provide data in Extensible Markup Language (XML) or other suitable formats. Although shown as an integrated component of computer 500, alternative implementations may show API 508 and / or service layer 510 as independent components relative to other components of computer 500. Furthermore, any or all portions of API 508 and / or service layer 510 may be implemented as a descendant or submodule of another software module, enterprise application, or hardware module without departing from the scope of this disclosure.

[0077] Computer 500 includes interface 502. Although in Figure 5 While shown as a single interface 502, two or more interfaces 502 may be used depending on the specific needs, expectations, or implementation of computer 500. Interface 502 is used by computer 500 to communicate with other systems (whether shown or not) in a distributed environment connected to a network. Generally, interface 502 includes logic encoded in software and / or hardware in a suitable combination and operable to communicate with the network. More specifically, interface 502 may include software supporting one or more communication protocols associated with the communication, enabling the network or interface hardware to transmit physical signals within and outside computer 500.

[0078] Computer 500 includes at least one processor 504. Although in Figure 5The processor 504 is shown as a single processor 504, but two or more processors may be used depending on the specific needs, expectations, or implementation of the computer. Generally, processor 504 executes instructions and manipulates data to perform operations of computer 500. Specifically, processor 504 executes... Figures 1-4 The publicly available functions.

[0079] Computer 500 also includes memory 514 for storing data for computer 500. Although in Figure 5 The memory 514 is shown as a single memory, but two or more memories may be used depending on the specific needs, expectations, or particular implementation of the computer 500. Although the memory 514 is shown as an integrated component of the computer 500, in alternative implementations, the memory 514 may be external to the computer 500.

[0080] Application 506 is an algorithmic software engine that provides functionality (particularly functionality required for anomaly detection) based on the specific needs, expectations, or implementation of computer 500. Although shown as a single application 506, application 506 can be implemented as multiple applications 506 on computer 500. Furthermore, although shown as integrated into computer 500, in alternative implementations, application 506 can be external to computer 500.

[0081] There may be any number of computers 500 associated with or outside the network and communicating through the network. Furthermore, this disclosure envisions that many users may use one computer 500, or one user may use multiple computers 500.

[0082] The subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in a combination of one or more of these. The software implementation of the described subject matter can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory computer-readable medium, for performing or controlling the operation of a computer or a computer-implemented system by a computer. Alternatively or additionally, the program instructions can be encoded in / on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a receiver device for execution by a computer or a computer-implemented system. Computer storage media can be machine-readable storage devices, machine-readable storage substrates, random or serial access memory devices, or combinations of computer storage media. Configuring one or more computers means that one or more computers have hardware, firmware, or software (or a combination of hardware, firmware, and software) installed such that when the software is executed by one or more computers, specific computational operations are performed. However, computer storage media are not propagation signals.

[0083] The terms “data processing apparatus,” “computer,” “computing device,” or “electronic computer equipment” (or equivalent terms as understood by one of ordinary skill in the art) refer to data processing hardware and encompass all kinds of apparatus, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A computer may also be or include special-purpose logic circuitry, such as a central processing unit (CPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some implementations, a computer or computer-implemented system or special-purpose logic circuitry (or a combination of a computer or computer-implemented system and special-purpose logic circuitry) may be hardware-based or software-based (or a combination of both). A computer may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of execution environments. This disclosure contemplates the use of a computer or computer-implemented system with an operating system (e.g., LINUX, UNIX, WINDOWS, MAC OS, ANDROID, or IOS) or a combination of operating systems.

[0084] A computer program (which may also be referred to or described as a program, software, software application, unit, module, software module, script, code, or other component) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form (including, for example, as a standalone program, module, component, or subroutine) for use in a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), stored in a single file dedicated to the program in question, or stored in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on a single computer, or on multiple computers located in one location or distributed among multiple locations and interconnected via a communication network.

[0085] While portions of the program shown in the various figures can be represented as separate components (such as units or modules) implementing the described features and functions using various objects, methods, or other processes, the program can be suitably modified to include multiple sub-units, sub-modules, third-party services, components, libraries, and other components. Conversely, the features and functions of various components can be suitably combined into a single component. The thresholds used for computational determination can be determined statically, dynamically, or both statically and dynamically.

[0086] The described methods, processes, or logical flows represent one or more examples of functionality consistent with this disclosure and are not intended to limit this disclosure to the implementations described or shown, but rather to conform to the widest scope consistent with the described principles and features. The described methods, processes, or logical flows can be executed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating output data. The methods, processes, or logical flows can also be executed by special-purpose logic circuitry (e.g., a CPU, FPGA, or ASIC), and the computer can also be implemented as special-purpose logic circuitry (e.g., a CPU, FPGA, or ASIC).

[0087] A computer used to execute computer programs can be based on a general-purpose microprocessor or a special-purpose microprocessor, both, or another CPU type. Generally, a CPU receives instructions and data from memory and writes instructions and data to memory. The basic components of a computer are a CPU for executing or running instructions and one or more storage devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, to receive data from, transfer data to, or both. However, a computer does not necessarily need to have such a device. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable memory storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0088] Non-transitory computer-readable media used for storing computer program instructions and data can include all forms of permanent / non-permanent or volatile / non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as random access memory (RAM), read-only memory (ROM), phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic devices, such as magnetic tape, magnetic tape cassette, cartridge magnetic tape, internal / removable magnetic disk; magneto-optical disk; optical memory devices, such as digital versatile / video optical disc (DVD), compressed optical disc (CD) ROM, DVD+ / -R, DVD-RAM, DVD-ROM, high definition / density (HD)-DVD, and BLU-RAY / BLU-RAY DISC (BD); and other optical memory technologies. Memory can store various objects or data, including: caches, categories, frames, applications, modules, backup data, jobs, web pages, web page templates, data structures, database tables, repositories storing dynamic information, or other suitable information including any parameters, variables, algorithms, instructions, rules, constraints, or references. Additionally, memory can include other suitable data such as logs, policies, security or access data, or report files. Processors and memory can be supplemented by special-purpose logic circuitry or incorporated into it.

[0089] To provide interaction with the user, the implementation of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), or plasma monitor) and a keyboard and pointing device (e.g., a mouse, trackball, or trackpad) that the user can use to provide input to the computer. Input can also be provided to the computer using a touchscreen (such as a pressure-sensitive tablet computer surface or a multi-touch screen using capacitive or electro-sensing). Other types of devices can be used to interact with the user. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, tactile feedback, or a combination of feedback types). Input from the user can be received in any form, including voice input, speech input, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the client computing device used by the user (e.g., by sending a webpage to a web browser on the user's mobile computing device in response to a request received from a web browser).

[0090] The term "graphical user interface (GUI)" can be used in the singular or plural to describe one or more graphical user interfaces and each display of a particular graphical user interface. Therefore, a GUI can represent any graphical user interface, including but not limited to web browsers, touchscreens, or command-line interfaces (CLIs) that process information and efficiently present the results to the user. Generally, a GUI may include multiple user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, dropdown lists, and buttons. These UI elements, and other UI elements, may be related to or represent the functionality of the web browser.

[0091] The implementations of the subject matter described in this specification can be implemented in a computing system including back-end components (e.g., as a data server), or in a computing system including middleware components (e.g., an application server), or in a computing system including front-end components (e.g., a client computer having a graphical user interface or web browser that a user can use to interact with the implementations of the subject matter described in this specification), or in any combination of one or more such back-end components, middleware components, or front-end components. Components of the system can be interconnected via any form or medium of wired or wireless digital data communication (or a combination of data communications), such as a communication network. Examples of communication networks include local area networks (LANs), radio access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), WiMAX, wireless local area networks (WLANs) using protocols such as 802.11x or others, all or part of the Internet, another communication network, or a combination of communication networks. Communication networks can communicate with, for example, Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, voice, video, data, or other information between network nodes.

[0092] A computing system may include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship arises from computer programs running on their respective computers that have a client-server relationship with each other.

[0093] In some implementations, any or all components (hardware and / or software) of the computing system can interface with each other and / or with interfaces using APIs and / or service layers. APIs may include specifications for routines, data structures, and object classes. APIs may be language-independent or language-dependent, and refer to complete interfaces, single functions, or even sets of APIs. Service layers provide software services to the computing system. The functionality of the various components of the computing system can be accessible to all service consumers via this service layer. Software services provide reusable, defined business functionality through defined interfaces. For example, an interface may be software written in JAVA, C++, or other suitable languages ​​that provide data in XML or other suitable formats. APIs and / or service layers may be integrated components and / or independent components relative to other components of the computing system. Furthermore, any or all portions of the service layer may be implemented as a descendant or submodule of another software module, enterprise application, or hardware module without departing from the scope of this disclosure.

[0094] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope or claimable scope of any inventive concept, but rather as descriptions of features characteristic of a particular implementation of a specific inventive concept. Certain features described in this specification within the context of individual implementations may also be implemented in combination within a single implementation. Conversely, various features described in the context of a single implementation may be implemented in multiple implementations, individually, or in any sub-combination. Furthermore, although the features described above may be described as functioning in certain combinations, and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0095] Specific implementations of the subject matter have been described. It will be apparent to those skilled in the art that other implementations, modifications, and substitutions of the described implementations are within the scope of the following claims. Although operations are depicted in a specific order in the drawings or claims, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or requiring the performance of all shown operations (some operations may be considered optional) to obtain the desired result. In some cases, multitasking or parallel processing (or a combination of multitasking and parallel processing) may be advantageous and may be implemented as appropriate.

[0096] The separation or integration of various system modules and components in the implementation methods described above should not be construed as requiring such separation or integration in all implementation methods. It should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0097] The implementation of the described subject may include one or more individual or combined features.

[0098] For example, in one implementation, a first feature of a method for modifying tags associated with files is proposed, where the tags are information that helps classify files, and the method includes: Obtain features from the file; Files are clustered based on a similarity measure between features, which is independent of the labels associated with the files; A mismatch between the identifier and the tag associated with a file belonging to a given cluster; Modify some tags of files in a given cluster based on the tag distribution in the given cluster.

[0099] The second feature, which can be combined with any of the preceding or following features, relates to a method for modifying the label associated with a file, wherein the modification includes assigning the same label to all files belonging to a given cluster.

[0100] The third feature, which can be combined with any preceding or following feature, relates to a method for modifying tags associated with a file, where the same tag corresponds to a tag that appears in the majority of a given cluster.

[0101] The fourth feature, which can be combined with any of the preceding or following features, relates to a method for modifying tags associated with a file, wherein the method also includes modifying the cluster before modifying some tags when the tag distribution in a given cluster is substantially equal in probability.

[0102] The fifth feature, which can be combined with any preceding or following feature, relates to a method for modifying tags associated with a file, wherein modifying clusters includes reducing the size of the clusters.

[0103] The sixth feature, which can be combined with any preceding or following feature, relates to a method for modifying tags associated with a document, wherein reducing the size of the cluster includes modifying the threshold associated with the similarity metric.

[0104] The seventh feature, which can be combined with any preceding or following feature, relates to a method for modifying labels associated with a document, wherein obtaining features from the document includes feeding the document to a trained model, wherein the trained model is an encoder model associated with a trained autoencoder model.

[0105] The eighth feature, which can be combined with any preceding or following feature, relates to a method for modifying tags associated with a document, wherein obtaining features from the document also includes processing the output from another trained model, wherein the other trained model has one of the following architectures: a convolutional neural network architecture, a transformer architecture.

[0106] The ninth feature, which can be combined with any of the preceding or following features, relates to a method for modifying the tags associated with a document, wherein clustering the documents involves applying a dimensionality reduction technique that relies on principal component analysis or t-distributed random neighborhood embedding (t-SNE).

[0107] The tenth feature, which can be combined with any of the preceding or following features, relates to a method for modifying the tags associated with a document, wherein clustering the documents involves applying one of the following to the feature: K-means clustering, hierarchical clustering, or density-based noisy spatial clustering application (DBSCAN).

[0108] The eleventh feature, which can be combined with any preceding or following feature, relates to a method for modifying labels associated with files, wherein clustering files includes: determining an HNSW graph, wherein each node of the HNSW graph corresponds to a feature associated with only one file, and wherein the HNSW graph is determined iteratively, adding nodes one by one, and wherein clustering files also includes obtaining a similarity matrix associated with the HNSW graph; and processing the similarity matrix into label clusters.

[0109] The twelfth feature, which can be combined with any preceding or following feature, relates to a method for modifying tags associated with a file, wherein obtaining the similarity matrix includes determining pairwise distances between all nodes in the HNSW graph, and wherein the distances are Euclidean distances or cosine similarity distances.

[0110] The thirteenth feature, which can be combined with any of the preceding features, relates to a method for modifying tags associated with a file, where the file is a binary file.

[0111] The example implementations described above do not define or constrain this disclosure. Other changes, substitutions, and modifications are possible without departing from the scope of this disclosure.

[0112] Furthermore, any claimed implementation is considered to be applicable at least to: a computer-implemented method; a non-transitory computer-readable medium storing computer-readable instructions for performing the computer-implemented method; and a computer system including a computer memory interoperably coupled to a hardware processor configured to perform the computer-implemented method or the instructions stored on the non-transitory computer-readable medium.

[0113] Finally, according to one embodiment, some machine learning models can run on a central processing unit (CPU), a general-purpose processor that handles most types of computational tasks. In one variation, a graphics processing unit (GPU) (specialized hardware designed for parallel computing) can be used to run or train the machine learning models mentioned in this document. Furthermore, in one variation, a tensor processing unit (TPU) can be used. Therefore, a device including at least one of these different processors can perform parts of the process involving the use of machine learning models.

Claims

1. A computer-implemented method for modifying tags associated with a file, wherein the tags are information that helps classify the file, wherein the method includes: Obtain features from the file; The files are clustered based on a similarity measure between the features, the similarity measure being independent of the tags associated with the files; A mismatch between the identifier and the tag associated with a file belonging to a given cluster; Modify some tags of files in the given cluster based on the tag distribution in the given cluster.

2. The computer-implemented method according to claim 1, wherein the modification includes: Assign the same label to all files belonging to the given cluster.

3. The computer-implemented method of claim 2, wherein the same label corresponds to a label that appears in a majority of the given cluster.

4. The computer-implemented method of claim 1, wherein when the label distributions in the given cluster are substantially equal in probability, the method further comprises: Modify the cluster before modifying some labels.

5. The computer-implemented method according to claim 4, wherein modifying the cluster includes: Reduce the size of the cluster.

6. The computer-implemented method of claim 5, wherein reducing the size of the cluster comprises: Modify the threshold associated with the similarity metric.

7. The computer-implemented method of claim 1, wherein obtaining features from the document comprises: The file is provided to a trained model, wherein the trained model is an encoder model associated with a trained autoencoder model.

8. The computer-implemented method of claim 7, wherein obtaining the features from the document further comprises: The output from the trained model is processed using another trained model, wherein the other trained model has one of the following architectures: a convolutional neural network architecture, a transformer architecture.

9. The computer-implemented method according to claim 1, wherein clustering the files comprises: The application relies on dimensionality reduction techniques such as principal component analysis or t-distributed random neighborhood embedding t-SNE.

10. The computer-implemented method of claim 1, wherein clustering the files comprises applying one of the following to the features: K-means clustering, hierarchical clustering, or density-based noisy spatial clustering application DBSCAN.

11. The computer-implemented method according to claim 1, wherein clustering the files comprises: The HNSW graph is determined, wherein each node of the HNSW graph corresponds to a feature associated with only one file, and wherein the HNSW graph is determined iteratively, adding nodes one by one, and wherein clustering the files further includes obtaining a similarity matrix associated with the HNSW graph, and processing the similarity matrix into identifier clusters.

12. The computer-implemented method of claim 11, wherein obtaining the similarity matrix comprises: Determine the pairwise distances between all nodes in the HNSW graph, where the distances are Euclidean distances or cosine similarity distances.

13. The computer-implemented method according to claim 1, wherein the file is a binary file.

14. A computer-readable medium containing instructions that, when executed, cause an electronic device to perform operations for modifying tags associated with a file, wherein the tags are information that helps classify the file, the operations including: Obtain features from the file; The files are clustered based on a similarity measure between the features, the similarity measure being independent of the tags associated with the files; A mismatch between the identifier and the tag associated with a file belonging to a given cluster; Modify some tags of files in the given cluster based on the tag distribution in the given cluster.

15. The computer-readable medium of claim 14, wherein the modification includes: Assign the same label to all files belonging to the given cluster.

16. A computer-implemented system, comprising: One or more computers; as well as One or more computer storage devices, interoperably coupled to the one or more computers, and having a tangible, non-transitory, machine-readable medium storing one or more instructions, which, when executed by the one or more computers, perform one or more operations for modifying tags associated with files, wherein the tags are information that helps classify files, and the one or more operations include: Obtain features from the file; The files are clustered based on a similarity measure between the features, the similarity measure being independent of the tags associated with the files; A mismatch between the identifier and the tag associated with a file belonging to a given cluster; Modify some tags of files in the given cluster based on the tag distribution in the given cluster.

Images