Machine Learning Architecture for Detecting Malicious Files Using Data Streams

Abstract

This application discloses a method, system, and computer system for classifying stream data at an edge device, the method including obtaining a stream of a file at an edge device, processing a set of chunks associated with the stream of the file using a machine learning model, and classifying the file at the edge device before processing the entire file.

Description

[Background technology]

[0001] Malicious individuals attempt to compromise computer systems in a variety of ways. As one example, such individuals may embed or otherwise include malicious files in email attachments and then send or have unsuspecting users send the malicious files. When executed, the malicious files compromise the victim's computer. Some types of malicious files instruct the compromised computer to communicate with a remote host. For example, a malicious file can turn a compromised computer into a "bot" in a "botnet," receiving instructions from and / or reporting data to a command and control (C&C) server under the malicious individual's control. One approach to mitigating the damage caused by malicious files is for a security company (or other appropriate entity) to identify the malicious file and attempt to prevent it from reaching and executing on end-user computers. Another approach is to prevent the compromised computer from communicating with the C&C server. Unfortunately, malicious file creators are using increasingly sophisticated techniques to obfuscate the workings of software, and therefore there is a continuing need for improved techniques to detect malware and prevent its harm. [Brief explanation of the drawings]

[0002] Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings. [Figure 1] FIG. 1 is a block diagram of an environment in which malicious traffic is detected or suspected, according to various embodiments. [Figure 2] FIG. 2 is a block diagram of a system for classifying files, according to various embodiments. [Figure 3] FIG. 3 is a block diagram of a method for classifying models. [Figure 4] FIG. 4 illustrates a system for classifying a streaming file based on a subset of chunks of the streaming file, according to various embodiments. [Figure 5] FIG. 5 illustrates a system for classifying a streaming file based on a subset of chunks of the streaming file, according to various embodiments. [Figure 6] FIG. 6 illustrates a graph of the performance of file classification using a subset of chunks of a streaming file, according to various embodiments. [Figure 7] FIG. 7 is a flowchart of a method for classifying a streaming file before processing the entire streaming file, according to various embodiments. [Figure 8] FIG. 8 is a flowchart of a method for classifying a streaming file before processing the entire streaming file, according to various embodiments. [Figure 9] FIG. 9 is a flowchart of a method for classifying a streaming file before processing the entire streaming file, according to various embodiments. [Figure 10] FIG. 10 is a flowchart of a method for training a classification model, according to various embodiments. [Figure 11] FIG. 11 is a diagram of a set of chunks associated with a streaming file, according to various embodiments. [Figure 12] FIG. 12 is a flowchart of a method for detecting malicious files, according to various embodiments. [Figure 13]FIG. 13 is a block diagram of a system for classifying streaming files based on chunk data, according to various embodiments. [Figure 14] FIG. 14 is a block diagram of a classification of a set of chunks captured within a stream of a file, according to various embodiments. [Figure 15] FIG. 15 is a block diagram of a classification of a set of chunks captured within a stream of a file, according to various embodiments. [Figure 16] FIG. 16 is a flowchart of a method for classifying a stream of files, according to various embodiments. [Figure 17] FIG. 17 is a flowchart of a method for detecting malicious files, according to various embodiments. [Figure 18] FIG. 18 is a flowchart of a method for detecting malicious files, according to various embodiments. [Figure 19] FIG. 19 is a flowchart of a method for detecting malicious files, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0003] The present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. These implementations, or any other form the present invention may take, may be referred to herein as techniques. In general, the order of steps in disclosed processes may be varied within the scope of the present invention. Unless otherwise specified, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0004] A detailed description of one or more embodiments of the present invention is provided below along with accompanying figures that illustrate the principles of the invention. While the present invention will be described in connection with such embodiments, the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For the purposes of clarity, known technical material in the art relevant to the present invention has not been described in detail so as not to unnecessarily obscure the present invention.

[0005] As used herein, an edge device may include a device (e.g., a hardware system) that controls data flow at the boundary between two networks. As one example, an edge device is a device that provides an entry point into an enterprise or service core network. Examples of edge devices include inline security entities, such as firewalls. Other examples of edge devices include routers, routing switches, integrated access devices, multiplexers, and wide area network access devices.

[0006] As used herein, an inline security entity may include a network node (e.g., a device) that enforces one or more security policies with respect to information such as network traffic, files, etc. As one example, a security entity may be a firewall. As another example, an inline security entity may be implemented as a router, a switch, a DNS resolver, a computer, a tablet, a laptop, a smartphone, etc. Various other devices may be implemented as a security entity. As another example, an inline security entity may be implemented as an application running on a device, such as an anti-malware application. As another example, an inline security entity may be implemented as an application running on a container or virtual machine.

[0007] Various embodiments include systems, methods, and devices for classification of streaming files. In some embodiments, classification of streaming files includes security processing at an inline security entity. The method includes acquiring a stream of the file at an edge device, processing a set of chunks associated with the stream of the file using a machine learning model, and classifying the file at the edge device before processing the entire file.

[0008] Various embodiments include systems, methods, and devices for classification of streaming files. In some embodiments, classification of streaming files includes security processing at an inline security entity. The method includes acquiring a stream of files at an edge device, aligning a predetermined amount of data within chunks associated with the stream of files, processing the aligned chunks associated with the stream of files using a machine learning model, and classifying the file at the edge device based at least in part on the classification of the aligned chunks.

[0009] Related art systems that classify files, including streaming files, perform classification after receiving the entire file. For example, related art systems classify files by using all (or substantially all) of the file to predict a classification for the file. File classification by related art systems may include performing feature extraction over the entire file (or substantially the entire file) and querying a model, such as a machine learning model, to obtain a file classification prediction (e.g., the likelihood that the file is malicious, etc.). As one example, a related art system used an XGBoost machine learning model to perform classification of non-streaming files on an edge device.

[0010] Related art systems generally are not feasible techniques for classifying streaming files because they must wait for the entire file to complete the transaction (e.g., be downloaded) before performing feature extraction on the streaming file. Because related art systems wait for the entire file to be received before performing classification (e.g., feature extraction and classification using a model), the related art systems are inefficient and introduce latency in consuming streaming data in the streaming file. Furthermore, using related art systems on edge devices is impractical due to memory constraints. Edge devices generally cannot store chunks of data (e.g., packets) locally, and therefore, portions of the streaming file are transferred to connected devices before the related art systems can identify the classification of the streaming file, such as whether the streaming file is malicious.

[0011] Various embodiments disclose systems, methods, and devices for performing classification on streaming files at an edge device (e.g., a firewall) and before the entire respective streaming file is processed at the edge device (e.g., before the entire streaming file is received). The system may perform classification of streaming files based at least in part on one or more chunks of the streaming file. As one example, a chunk may be a predetermined number of bytes of data (e.g., 1500 bytes of data). In some embodiments, the system analyzes each chunk sequentially (e.g., upon receipt of the chunk) and performs a classification prediction on the streaming file before the entire streaming file is received / processed. In response to a particular classification of a chunk of the streaming file (e.g., if the prediction that the file corresponds to a particular classification exceeds a predetermined classification threshold), the system may perform an active measure on the streaming file.

[0012] For classification in the context of detecting malicious files, the system processes the chunks sequentially and allows the chunk to pass through the system (e.g., be executed by the device) if the chunk does not indicate a malicious file (e.g., the file is not classified as malicious based on the chunk), and takes active action with respect to further chunks if the chunk indicates that the streaming file is malicious (e.g., the file is classified as malicious based on the chunk). One example of an active action could be blocking the remaining chunks of the streaming file from passing through (or being processed by) the edge device.

[0013] In some embodiments, the system uses a machine learning model trained using streamlined deep learning techniques to facilitate classification of streaming files at the edge device. The machine learning model is trained to classify files a chunk at a time (e.g., sequentially classify a predetermined number of bytes of data). Due to severe memory constraints on the edge device, storing the entire file is impractical. However, various embodiments save some state information that indicates the state of the streaming file. The state information is used to classify the current chunk, and the system then iterates between saving the state information and using such information to classify the next chunk. In some embodiments, the state information corresponds to the results of a max pooling operation performed on a subset of the streaming file (e.g., one or more chunks of the streaming file).

[0014] In some cases, the profile of a file received at an edge device is non-linear. For example, a given file type has header information contained in the first chunk (e.g., first packet). However, for classification of a streaming file based on chunk-level classification (e.g., analysis / prediction using a single chunk at a time until completion) to be deterministic, the classifier (e.g., machine learning model) must be analyzing the same type of bytes (e.g., bytes containing non-header information) at any given time. Various embodiments implement alignment of the chunks of a streaming file in relation to ensuring that classification is performed on the same set of bytes.

[0015] Various embodiments improve upon related art systems because streaming files are classified at an edge device and classification can be performed before the entire streaming file is received or processed. Thus, various embodiments enable a system to take action regarding a streaming file based on its classification sooner, before the entire streaming file is received or processed.

[0016] Although the embodiments described in connection with the examples illustrated in Figures 1-19 are described primarily in the context of malicious file / traffic detection (e.g., classifying a file as malicious / non-malicious based on analysis of a subset of the file's chunks), various embodiments may be implemented in other contexts for classifying streaming files. Examples of other contexts include, but are not limited to, classifying files as containing / associated with financial information, HIPPA information, personally identifiable information (PII), copyrighted material, General Data Protection Regulation (GDPR) data, etc. As one example, various embodiments classify (or predict) a streaming file as containing copyrighted material based on analysis of a subset of the streaming file's chunks (e.g., before the entire streaming file is received or processed).

[0017] 1 is a block diagram of an environment in which malicious traffic is detected or suspected, according to various embodiments. In the illustrated example, client devices 104-108 are a laptop computer, a desktop computer, and a tablet (respectively) residing within a corporate network 110 (belonging to "Acme Company"). Data appliance 102 (e.g., an edge device) is configured to enforce policies (e.g., security policies) regarding communications between client devices, such as client devices 104 and 106, and nodes outside corporate network 110 (e.g., reachable via external network 118). Examples of such policies include those governing traffic shaping, quality of service, and traffic routing. Other examples of policies include security policies requiring scanning for threats in incoming (and / or outgoing) email attachments, website content, input to application portals (e.g., web interfaces), files exchanged via instant messaging programs, and / or other file transfers. In some embodiments, the data appliance 102 is also configured to enforce policies regarding traffic leaving (or entering) the enterprise network 110. For example, the data appliance 102 also enforces policies regarding leakage or inappropriate transmission of certain data, such as GDPR data, PII, etc.

[0018] In the illustrated example, the data appliance 102 is an inline security entity. However, various other implementations may include a data appliance that is another type of edge device (e.g., a device that does not specifically provide inline security processing). The data appliance 102 performs low-latency processing / analysis of received data (e.g., traffic data) and determines whether to offload any processing of the received data to a cloud system, such as security platform 140. As one example, the data appliance 102 processes streaming files and classifies the streaming files locally. In some embodiments, the data appliance 102 classifies streaming files based on a subset of the streaming data before the entirety of each streaming file is received / processed. For example, the data appliance 102 may perform classification using individual chunks (e.g., packets or a predetermined number of bytes). In connection with performing classification using individual chunks, the data appliance sequentially performs feature extraction on the chunks and classifies the streaming file based at least in part on the feature extraction, and then continues to iteratively perform such analysis on a chunk-by-chunk basis (e.g., in the order in which the chunks are received) until the earlier of (i) the streaming file is classified (e.g., a prediction obtained based on the classification exceeds a predefined threshold, such as a predefined maliciousness threshold), and (ii) the streaming file is completely received or processed. For example, the data appliance 102 queries a classifier or model (e.g., a machine learning model) stored locally on the data appliance 102 based at least in part on the feature extraction of a particular chunk to obtain a classification prediction for the streaming file using the chunk.

[0019] The techniques described herein may be used in conjunction with various platforms (e.g., desktops, mobile devices, gaming platforms, embedded systems, etc.) and / or various types of applications (e.g., Android .apk files, iOS applications, Windows PE files, Adobe Acrobat PDF files, Microsoft Windows PE installers, etc.). In the example environment shown in Figure 1, client devices 104-108 are a laptop computer, a desktop computer, and a tablet (respectively) that reside within enterprise network 110. Client device 120 is a laptop computer that resides outside enterprise network 110.

[0020] The data appliance 102 may be configured to operate in cooperation with a remote security platform 140. The security platform 140 may be a cloud system such as a cloud services security entity. The security platform 140 may provide various services, including performing static and dynamic analysis on malware samples, providing a list of signatures of known exploits (e.g., malicious input strings, malicious files, etc.) to a data appliance such as the data appliance 102 as part of a subscription, detecting exploits such as malicious input strings or malicious files (e.g., on-demand detection or periodic updates to the mapping of input strings or files to an indication of whether the input string or file is malicious or benign), providing the likelihood that an input string or file is malicious or benign, providing / updating a whitelist of input strings or files that are considered benign, providing / updating input strings or files that are considered malicious, identifying malicious input strings, detecting malicious input strings, detecting malicious files, predicting whether an input string or file is malicious, and providing an indication that an input string or file is malicious (or benign). In various embodiments, the results of the analysis (and additional information about the application, domain, etc.) are stored in database 160. In various embodiments, security platform 140 includes one or more dedicated, commercially available hardware servers (e.g., having multi-core processors, 32G or more RAM, gigabit network interface adapters, and hard drives) running a typical server-class operating system (e.g., Linux).Security platform 140 may be implemented across a scalable infrastructure, including multiple such servers, solid-state drives, and / or other applicable high-performance hardware. Security platform 140 may comprise several distributed components, including components provided by one or more third parties. For example, some or all of security platform 140 may be implemented using Amazon Elastic Compute Cloud (EC2) and / or Amazon Simple Storage Service (S3). Furthermore, similar to data appliance 102, whenever security platform 140 is referred to as performing a task, such as storing or processing data, it should be understood that one or more subcomponents of security platform 140 may perform that task collaboratively (either individually or in cooperation with third-party components). As one example, security platform 140 may optionally collaborate with one or more virtual machine (VM) servers to perform static / dynamic analysis. An example of a virtual machine server is a physical machine that includes commercially available server-class hardware (e.g., a multi-core processor, 32 gigabytes or more of RAM, and one or more gigabit network interface adapters) running commercially available virtualization software such as VMware ESXi, Citrix XenServer, or Microsoft Hyper-V. In some embodiments, the virtual machine server is omitted. Furthermore, the virtual machine server may be under the control of the same entity that manages security platform 140, but may also be provided by a third party. As one example, the virtual machine server may rely on EC2, with the remainder of security platform 140 being provided by dedicated hardware owned and under the control of the operator of security platform 140.

[0021] In some embodiments, system 100 uses security platform 140 to perform processing on traffic data offloaded by data appliance 102, such as performing computationally intensive processing. Security platform 140 provides one or more services to data appliance 102, client devices 120, etc. Examples of services provided by security platform 140 (e.g., a cloud service entity) include a data loss prevention (DLP) service, an application cloud engine (ACE) service (e.g., a service for identifying application types based on traffic patterns or fingerprints), a machine learning command and control (MLC2) service, an advanced URL filtering (AUF) service, a threat detection service, an enterprise data leak service (e.g., detecting data leaks or identifying the source of a leak), and an Internet of Things (IoT) service. Various other services may be implemented.

[0022] In some embodiments, system 100 (e.g., malicious sample detector 170, security platform 140, etc.) trains detection models to detect exploits (e.g., malicious samples), malicious traffic, application identification information, or to detect certain types of information (e.g., predefined categories of information such as financial information, GDPR data, PII, etc.). Security platform 140 may store blacklists, whitelists, etc. related to data (e.g., mappings of signatures to malicious files, etc.). In response to processing the traffic data, security platform 140 may send updates to inline security entities, such as data appliance 102. For example, security platform 140 provides updates to mappings of signatures to malicious files, updates to mappings of signatures to benign files, etc.

[0023] According to various embodiments, models trained by system 100 (e.g., security platform 140) are obtained using a machine learning process. Examples of machine learning processes that may be implemented in connection with training a model include random forests, linear regression, support vector machines, naive Bayes, logistic regression, K-nearest neighbors, decision trees, gradient-boosted decision trees, K-means clustering, hierarchical clustering, density-based spatial clustering of noisy applications (DBSCAN) clustering, principal component analysis, etc. In some embodiments, the system trains an XGBoost machine learning classifier model. As one example, the input to the classifier (e.g., an XGBoost machine learning classifier model) is a combined feature vector or set of feature vectors, and based on the combined feature vector or set of feature vectors, the classifier model determines whether corresponding traffic (e.g., an input string) is malicious or the likelihood that the traffic is malicious (e.g., whether the traffic is exploit traffic).

[0024] According to various embodiments, security platform 140 includes DNS tunneling detector 138 and / or malicious sample detector 170. Malicious sample detector 170 is used in connection with determining whether a sample (e.g., traffic data) is malicious. In response to receiving a sample (e.g., an input string, such as an input string entered in connection with a login attempt, a file, or a traffic pattern), malicious sample detector 170 analyzes the sample (e.g., the input string, etc.) and determines whether the sample is malicious. For example, malicious sample detector 170 determines one or more feature vectors (e.g., combined feature vectors) for the sample and uses a model to determine (e.g., predict) whether the sample is malicious. Malicious sample detector 170 determines whether the sample is malicious based at least in part on one or more attributes of the sample. In some embodiments, malicious sample detector 170 receives a sample, performs feature extraction (e.g., feature extraction on one or more attributes of the input string), and determines (e.g., predicts) whether the sample (e.g., an SQL or command injection string) is malicious based at least in part on the feature extraction results. For example, malicious sample detector 170 uses a classifier (e.g., a detection model) to determine (e.g., predicts) whether the sample is malicious based at least in part on the feature extraction results. In some embodiments, the classifier corresponds to a model (e.g., a detection model) for determining whether a sample is malicious, and the model is trained using a machine learning process.

[0025] In some embodiments, the malicious sample detector 170 includes one or more of a traffic parser 172 , a prediction engine 174 , an ML model 176 , and / or a cache 178 .

[0026] The traffic parser 172 is used in connection with determining (e.g., isolating) one or more attributes associated with the sample being analyzed. As one example, in the case of a file, the traffic parser 172 can parse / extract information from the file, such as from the file's header. Information obtained from the file may include libraries, functions, or files invoked / called by the file being analyzed, the order of calls, etc. As another example, in the case of an input string, the traffic parser 172 determines a set of alphanumeric characters or values ​​associated with the input string. In some embodiments, the traffic parser 172 obtains one or more attributes associated with (e.g., from) the sample. For example, the traffic parser 172 obtains one or more patterns (e.g., alphanumeric patterns), one or more sets of alphanumeric characters, one or more commands, one or more pointers or links, one or more IP addresses, regular expression (regex) statements, etc. from the sample.

[0027] In some embodiments, one or more feature vectors corresponding to a sample are determined by malicious sample detector 170 (e.g., traffic parser 172 or prediction engine 174). For example, the one or more feature vectors are determined (e.g., populated) based at least in part on one or more characteristics or attributes associated with the sample (e.g., in the case where the sample is an input string, one or more attributes or sets of alphanumeric characters or values ​​associated with the input string). As one example, traffic parser 172 uses one or more attributes associated with the sample in connection with determining the one or more feature vectors. In some implementations, traffic parser 172 determines a combined feature vector based at least in part on one or more feature vectors corresponding to the sample. As one example, the set of one or more feature vectors is determined (e.g., configured or defined) based at least in part on a model used to detect exploits. Malicious sample detector 170 can use the set of one or more feature vectors to determine one or more attributes of a pattern used in connection with training or implementing a model (e.g., attributes whose fields are populated into the feature vector, etc.). The model may be trained using a set of features obtained at least in part based on sample malicious traffic, such as a set of features corresponding to predefined regular expression statements and / or a set of feature vectors determined based on algorithm-based feature extraction. For example, the model may be determined at least in part based on performing malicious feature extraction in connection with generating (e.g., training) a model for detecting exploits. The malicious feature extraction may include one or more of: (i) using predefined regular expression statements to obtain specific features from files or SQL and command injection strings; and (ii) using algorithm-based feature extraction to filter the described features from a set of raw input data.

[0028] In response to malicious sample detector 170 receiving a sample for which it is to determine whether the sample is malicious (or the likelihood that the sample is malicious), malicious sample detector 170 determines one or more feature vectors (e.g., individual feature vectors corresponding to a set of predefined regular expression statements, individual feature vectors corresponding to attributes or patterns obtained using algorithm-based analysis of the exploit, and / or a combined feature vector of both, etc.). As one example, in response to determining (e.g., obtaining) one or more feature vectors, malicious sample detector 170 (e.g., traffic parser 172) provides (or makes accessible) the one or more feature vectors to prediction engine 174 (e.g., in connection with obtaining a prediction of whether the sample is malicious). As another example, malicious sample detector 170 (e.g., traffic parser 172) stores the one or more feature vectors, such as in cache 178 or database 160.

[0029] In some embodiments, prediction engine 174 determines whether a sample is malicious based at least in part on one or more of: (i) a mapping of samples to an indication of whether the corresponding sample is malicious; (ii) a mapping of an identifier for the sample (e.g., a hash or other signature associated with the sample) to an indication of whether the corresponding sample is malicious; and / or (iii) a classifier (e.g., a model trained using a machine learning process). In some embodiments, determining whether a sample is malicious (e.g., based on a mapping of an identifier to an indication that the sample is malicious) may be performed in data appliance 102, and for samples for which an associated identifier is not stored in the mapping, data appliance 102 offloads processing of the sample to security platform 140.

[0030] The prediction engine 174 is used to predict whether a sample is malicious. In some embodiments, the prediction engine 174 determines (e.g., predicts) whether a received sample is malicious. The prediction engine 174 determines whether a newly received sample is malicious based at least in part on characteristics / attributes associated with the sample (e.g., regular expression statements, information obtained from a file header, calls to libraries, APIs, etc.). For example, the prediction engine 174 applies a machine learning model to determine whether a newly received sample is malicious. Applying the machine learning model to determine whether a sample is malicious may include the prediction engine 174 querying the machine learning model 176 (e.g., with information about the sample, one or more feature vectors, etc.). In some implementations, the machine learning model 176 is pre-trained, and the prediction engine 174 does not need to provide a set of training data (e.g., malicious traffic samples and / or benign traffic samples) to the machine learning model 176 at the same time as the query for an indication / determination of whether a particular sample is malicious. In some embodiments, prediction engine 174 receives information associated with whether a sample is malicious (e.g., an indication that the sample is malicious). For example, prediction engine 174 receives the results of a determination or analysis by machine learning model 176. In some embodiments, prediction engine 174 receives an indication of the likelihood that the sample is malicious from machine learning model 176. In response to receiving the indication of the likelihood that the sample is malicious, prediction engine 174 determines (e.g., predicts) whether the sample is malicious based at least in part on the likelihood that the sample is malicious. For example, prediction engine 174 compares the likelihood that the sample is malicious to a likelihood threshold (e.g., a pre-defined maliciousness threshold).In response to determining that the likelihood that the sample is malicious is greater than the likelihood threshold, the prediction engine 174 may consider (e.g., determine) the sample to be malicious. Conversely, in response to determining that the sample is malicious is greater than the likelihood threshold, the prediction engine 174 may consider (e.g., determine) the sample to be benign (e.g., non-malicious).

[0031] According to various embodiments, in response to the prediction engine 174 determining that the received sample is malicious, the security platform 140 transmits an indication that the sample is malicious to a security entity (e.g., the data appliance 102). For example, the malicious sample detector 170 may transmit an indication that the sample is malicious to an inline security entity (e.g., a firewall) or a network node (e.g., a client). The indication that the sample is malicious may correspond to an update to a blacklist of samples (e.g., corresponding to malicious samples), such as when the received sample is deemed malicious, or to an update to a whitelist of samples (e.g., corresponding to non-malicious samples), such as when the received sample is deemed benign. In some embodiments, the malicious sample detector 170 transmits a hash or signature corresponding to the sample in association with an indication of whether the sample is malicious or benign. A security entity or endpoint can compute a hash or signature for the sample and then perform a lookup (e.g., query a whitelist and / or blacklist) for a mapping of the hash / signature to an indication of whether the sample is malicious / benign. In some embodiments, the hash or signature uniquely identifies the sample.

[0032] The prediction engine 174 is used in connection with determining whether a sample (e.g., an input string) is malicious (e.g., determining the likelihood or prediction of whether the sample is malicious). The prediction engine 174 uses information associated with the sample (e.g., one or more attributes, patterns, etc.) in connection with determining whether the corresponding sample is malicious.

[0033] In response to receiving a sample to be analyzed, malicious sample detector 170 can determine whether the sample corresponds to a previously analyzed sample (e.g., whether the sample matches a sample associated with historical information for which a malicious determination was previously calculated). As one example, malicious sample detector 170 determines whether an identifier or representative information corresponding to the sample is included in historical information (e.g., a blacklist, a whitelist, etc.). In some embodiments, the representative information corresponding to the sample is a hash or signature of the sample. In some embodiments, malicious sample detector 170 (e.g., prediction engine 174) determines whether information related to a particular sample is included in a dataset of historical input strings and historical information (e.g., VirusTotal) associated with the historical dataset that indicates whether a particular sample is malicious. TMIn response to determining that information related to a particular sample is not included in or available from a dataset of historical input strings and historical information, malicious sample detector 170 may consider the sample to have not yet been analyzed, and may invoke analysis (e.g., dynamic analysis) of the sample in connection with determining (e.g., predicting) whether the sample is malicious (e.g., malicious sample detector 170 may query a classifier based on the sample in connection with determining whether the sample is malicious). One example of historical information associated with a historical sample that indicates whether a particular sample is malicious corresponds to a VirusTotal® (VT) score. A VT score greater than 0 for a particular sample indicates that the particular sample is deemed malicious by the third-party service. In some embodiments, historical information associated with a historical sample that indicates whether a particular sample is malicious corresponds to a social score, such as a community-based score or rating (e.g., a reputation score), that indicates that the sample is malicious or likely to be malicious. Historical information (e.g., from third-party services, community-based scores, etc.) indicates whether other vendors or cybersecurity organizations consider a particular sample to be malicious.

[0034] In some embodiments, malicious sample detector 170 (e.g., prediction engine 174) determines that a received sample is newly analyzed (e.g., the sample is not in historical information / datasets, is not on a whitelist or blacklist, etc.). Malicious sample detector 170 (e.g., traffic parser 172) may detect that a sample is newly analyzed in response to security platform 140 receiving the sample from a security entity (e.g., a firewall) or endpoint in the network. For example, malicious sample detector 170 determines that a sample is newly analyzed contemporaneously with receipt of the sample by security platform 140 or malicious sample detector 170. As another example, malicious sample detector 170 (e.g., prediction engine 174) determines that a sample is newly analyzed according to a predefined schedule (e.g., daily, weekly, monthly, etc.), such as associated with a batch process. In response to determining that the received sample has not yet been analyzed for maliciousness (e.g., the system does not contain historical information regarding such input string), the malicious sample detector 170 determines whether to use analysis (e.g., dynamic analysis) of the sample in connection with determining whether the sample is malicious (e.g., query a classifier to analyze the sample or one or more feature vectors associated with the sample), and the malicious sample detector 170 uses a classifier with respect to a characteristic within the sample, or a set of feature vectors or combined feature vectors associated with attribute or characteristic relationships.

[0035] The machine learning model 176 predicts whether a sample (e.g., a newly received sample) is malicious or not based at least in part on the model. As one example, the model is pre-stored and / or pre-trained. The model may be trained using various machine learning processes. According to various embodiments, the machine learning model 176 uses attributes, characteristics, relationships between attributes or characteristics, and / or patterns for the samples and / or training set to infer whether a sample is malicious or not, such as predicting the likelihood that a sample is malicious. For example, the machine learning model 176 uses a machine learning process to analyze a set of relationships between an indication of whether a sample is malicious (or benign) and one or more attributes associated with the sample, and then uses the set of relationships to generate a predictive model for predicting whether a particular sample is malicious or not. In some embodiments, in response to predicting that a particular sample is malicious, an association between the sample and the indication that the sample is malicious is stored in the malicious sample detector 170 (e.g., in the cache 178), etc. In some embodiments, in response to predicting the likelihood that a particular sample is malicious, an association between the sample and the likelihood that the sample is malicious is stored in malicious sample detector 170 (e.g., cache 178), etc. Machine learning model 176 may provide an indication of whether the sample is malicious or the likelihood that the sample is malicious to prediction engine 174. In some implementations, machine learning model 176 provides an indication to prediction engine 174 that analysis by machine learning model 176 is complete and that the corresponding results (e.g., predicted results) have been stored in cache 178.

[0036] Cache 178 stores information about samples (e.g., input strings). In some embodiments, cache 178 stores a mapping of a particular input string to an indication of whether the input string is (or may be) malicious, or a mapping of an indication of whether the sample is (or may be) malicious to a hash or signature corresponding to the sample. Cache 178 may store additional information related to a set of samples, such as attributes of the sample, hashes or signatures corresponding to samples within the set of samples, other unique identifiers corresponding to samples within the set of samples, etc. In some embodiments, an inline security entity, such as data appliance 102, stores a cache corresponding to or similar to cache 178. For example, the inline security entity may use a local cache to perform inline processing of traffic data, such as low-latency processing.

[0037] Returning to FIG. 1 , assume that a malicious individual (using client device 120) has created malware or malicious input string 130. The malicious individual hopes that a client device, such as client device 104, will execute a copy of the malware or other exploit (e.g., the malware or malicious input string) 130, compromising the client device and causing the client device to become a bot in a botnet. The compromised client device can then be instructed to perform a task (e.g., cryptocurrency mining or participating in a denial-of-service attack) and / or report information to an external entity, such as command and control (C&C) server 150 (e.g., associated with such task, exfiltrate sensitive corporate data, etc.), and, if applicable, receive instructions from C&C server 150.

[0038] The environment shown in Figure 1 includes three Domain Name System (DNS) servers (122-126). As shown, DNS server 122 is under the control of ACME (for use by computing assets located within enterprise network 110), while DNS server 124 is publicly accessible (and may also be used by computing assets located within network 110 as well as other devices, such as those located within other networks (e.g., networks 114 and 116)). DNS server 126 is publicly accessible but is under the control of malicious operators of C&C server 150. Enterprise DNS server 122 is configured to resolve enterprise domain names to IP addresses and, if applicable, is further configured to communicate with one or more external DNS servers (e.g., DNS servers 124 and 126) to resolve domain names.

[0039] To connect to a legitimate domain (e.g., www.example.com, shown as website 128), a client device, such as client device 104, needs to resolve the domain to a corresponding Internet Protocol (IP) address. One way such resolution can occur is for client device 104 to forward a request to DNS server 122 and / or 124 to resolve the domain. In response to receiving a valid IP address for the requested domain name, client device 104 can connect to website 128 using the IP address. Similarly, to connect to malicious C&C server 150, client device 104 needs to resolve the domain “kj32hkjqfeuo32ylhkjshdflu23.badsite.com” to a corresponding Internet Protocol (IP) address. In this example, malicious DNS server 126 is authoritative for *.badsite.com, and client device 104's request is forwarded to (for example) DNS server 126 for resolution, ultimately allowing C&C server 150 to receive data from client device 104.

[0040] Data appliance 102 is configured to enforce policies regarding communications between client devices, such as client devices 104 and 106, and nodes outside enterprise network 110 (e.g., reachable via external network 118). Examples of such policies include those governing traffic shaping, quality of service, and traffic routing. Other examples of policies include security policies, such as those requiring scanning for threats in incoming (and / or outgoing) email attachments, website content, information entered into web interfaces such as login screens, files exchanged via instant messaging programs, and / or other file transfers, and / or quarantining or deleting files or other exploits identified as malicious (or likely to be malicious). In some embodiments, data appliance 102 is also configured to enforce policies regarding traffic remaining within enterprise network 110. In some embodiments, the security policy includes instructions that network traffic (e.g., all network traffic, certain types of network traffic, etc.) should be classified / scanned by classifiers stored in a local cache, or that certain detected network traffic should otherwise be further analyzed (e.g., using finer-grained detection models), such as by offloading processing to security platform 140.

[0041] In various embodiments, the data appliance 102 includes a DNS module 134 configured to facilitate determining whether a client device (e.g., a client device 104-108) is attempting to engage in malicious DNS tunneling and / or to prevent connections (e.g., by a client device 104-108) to malicious DNS servers. The DNS module 134 may be integrated into the data appliance 102 (as shown in FIG. 1) or, in various embodiments, may operate as a standalone appliance. Like the other components shown in FIG. 1, the DNS module 134 may be provided by the same entity that provides the data appliance 102 (or security platform 140), or may be provided by a third party (e.g., different from the provider of the data appliance 102 or security platform 140). Furthermore, in addition to preventing connections to malicious DNS servers, the DNS module 134 can take other actions, such as individualized logging of tunneling attempts made by clients (indicating that a given client is at risk and should be quarantined or otherwise investigated by an administrator).

[0042] In various embodiments, when a client device (e.g., client device 104) attempts to resolve a domain, DNS module 134 uses the domain as a query to security platform 140. This query may be performed simultaneously with the resolution of the domain (e.g., with a request sent to DNS servers 122, 124, and / or 126 and security platform 140). As one example, DNS module 134 may send a query (e.g., in JSON format) to front end 142 of security platform 140 via a REST API. Using a process described in more detail below, security platform 140 determines (e.g., using DNS tunneling detector 138, such as decision engine 152 of DNS tunneling detector 138) whether the queried domain indicates a malicious DNS tunneling attempt and returns a result (e.g., “malicious DNS tunneling” or “non-tunneling”) to DNS module 134.

[0043] In various embodiments, when a client device (e.g., client device 104) attempts to resolve an SQL statement, SQL command, or other command injection string, the data appliance 102 uses the corresponding sample (e.g., input string) as a local cache and / or query to security platform 140. This query may be performed simultaneously with the resolution of the SQL statement, SQL command, or other command injection string. As one example, the data appliance 102 sends the query (e.g., in JSON format) to the front end 142 of the security platform 140 via a REST API. As another example, the data appliance 102 sends the query to the security platform 140 (e.g., the front end 142 of the security platform 140) directly from the data plane of the data appliance 102. For example, a process running on the data appliance 102 (e.g., a daemon, such as WIFClient, running on the data plane to facilitate offloading of processing data) communicates the query (e.g., a request message) to the security platform 140. The query is not first communicated to the message plane of the data appliance 102, which then communicates the query to the security platform 140. For example, the data appliance 102 is configured to use a process running on the data plane to query the security platform 140 without intermediation through the management plane of the data appliance 102. Using a process described in more detail below, the security platform 140 determines (e.g., using malicious sample detector 170) whether the queried SQL statement, SQL command, or other command injection string indicates an exploit attempt and provides a result (e.g., "malicious exploit" or "benign traffic") to the data appliance 102.

[0044] In various embodiments, when a client device (e.g., client device 104) attempts to open or receives a file or input string via email, an attachment to an instant message, or otherwise exchanged over a network, DNS module 134 uses the file or input string (or a calculated hash or signature, or other unique identifier, etc.) as a query to security platform 140. This query may be performed simultaneously with receipt of the file or input string or in response to a request from a user to scan the file. As one example, data appliance 102 may send a query (e.g., in JSON format) to front end 142 of security platform 140 via a REST API. The query may be communicated to the security platform by a process / connector implemented on the data plane of data appliance 102. Using processing described in more detail below, security platform 140 determines whether the queried file is malicious (or likely to be malicious) (e.g., using a malicious file detector, which may be similar to malicious sample detector 170, e.g., using a machine learning model to detect / predict whether a file is malicious) and provides a result (e.g., “malicious file” or “benign file”) back to data appliance 102.

[0045] In various embodiments, the DNS tunneling detector 138 (whether implemented in the security platform 140, the data appliance 102, or another suitable location / combination of locations) uses a two-pronged approach in identifying malicious DNS tunneling. The first approach uses the anomaly detector 146 (e.g., implemented using Python) to build a set of real-time profiles (156) of DNS traffic for root domains. The second approach uses signature generation and matching (also referred to herein as similarity detection, and implemented using Go, for example). The two approaches are complementary. The anomaly detector functions as a general-purpose detector that can identify previously unknown tunneling traffic. However, the anomaly detector may need to observe multiple DNS queries before a detection can be made. To block the first DNS tunneling packet, the similarity detector 144 complements the anomaly detector 146 and extracts signatures from the detected tunneling traffic that can be used to identify situations where an attacker has registered a new malicious tunneling root domain, but has done so using tools / malware that resemble the detected root domain.

[0046] When data appliance 102 receives DNS queries (e.g., from DNS module 134), data appliance 102 provides them to security platform 140, which performs both anomaly detection and similarity detection, respectively. In various embodiments, a domain (e.g., provided in a query received by security platform 140) is classified as a malicious DNS tunneling root domain if either detector flags the domain.

[0047] DNS tunneling detector 138 maintains, for each appliance (for which data is received), a set of fully qualified domain names (FQDNs) grouped by their root domain (collectively illustrated in FIG. 1 as domain profile 156). While grouping by root domain is generally described herein, it should be understood that the techniques described herein can be extended to any level of domain. In various embodiments, information about received queries for a given domain is persisted in a profile for a fixed amount of time (e.g., a sliding time window of 10 minutes).

[0048] As one example, DNS query information received from data appliance 102 for various foo.com sites is grouped (into a domain profile for the root domain foo.com) as follows: G(foo.com) = [mail.foo.com, coolstuff.foo.com, domain1234.foo.com]. A second root domain has a second profile with similar application information (e.g., G(baddomain.com) = [lskjdf23r.baddomain.com, kj235hdssd233.baddomain.com]). Each root domain (e.g., foo.com or baddomain.com) is modeled with a set of characteristics specific to malicious DNS tunneling, so that even diverse benign DNS patterns (e.g., k2jh3i8y35.legitimatesite.com, xxx888222000444.otherlegitimatesite.com) are highly unlikely to be misclassified as malicious tunneling. The following are example characteristics that may be extracted as features (eg, into a feature vector) for a given group of domains (ie, sharing a root domain):

[0049] In some embodiments, malicious sample detector 170 provides an indication of whether a sample is malicious to a security entity, such as data appliance 102. For example, in response to determining that a sample is malicious, malicious sample detector 170 sends an indication that the sample is malicious to data appliance 102, which may then implement one or more security policies based at least in part on the indication that the sample is malicious. The one or more security policies may include quarantining / blocking the entered string or file, deleting the sample, ensuring that the sample is not executed or resolved, warning or prompting a user of the sample's malicious intent before opening / executing the sample, etc. As another example, in response to determining that a sample is malicious, malicious sample detector 170 provides to the security entity an update to a mapping of the sample (or a hash, signature, or other unique identifier corresponding to the sample) to an indication of whether the corresponding sample is malicious, or an update to a blacklist of malicious samples (e.g., identifying the samples) or a whitelist of benign samples (e.g., identifying samples that are not considered malicious).

[0050] In some embodiments, one or more feature vectors corresponding to a sample, such as a file or an input string, are determined by the system 100 (e.g., the security platform 140, the malicious sample detector 170, the pre-filter 135, etc.). For example, the one or more feature vectors are determined (e.g., populated) based at least in part on one or more characteristics or attributes associated with the sample (e.g., if the sample is an input string, one or more attributes or sets of alphanumeric characters or values ​​associated with the input string). As one example, the system 100 uses features associated with a classifier (e.g., a machine learning model 176, such as a detection model) of the malicious sample detector 170 in connection with determining the one or more feature vectors, such as one or more attributes associated with the sample. In some implementations, the pre-filter 135 determines a combined feature vector based at least in part on one or more feature vectors corresponding to the sample. As one example, a set of one or more feature vectors is determined (e.g., set or defined) based at least in part on a pre-filter model (e.g., based on pre-filter features). The system 100 (e.g., the pre-filter 135) can use the set of one or more feature vectors to determine one or more attributes of the pattern (e.g., the attributes whose fields are populated in the feature vectors, etc.) used in connection with training or implementing the model. The pre-filter model can be trained using a set of features obtained based at least in part on the set of features used in connection with obtaining the detection model.

[0051] According to various embodiments, an edge device (e.g., an inline security entity such as data appliance 102) receives traffic data, such as a file, and locally classifies the traffic data. The edge device may use a local classifier (e.g., a machine learning model) stored in a cache, etc. For example, the edge device locally performs feature extraction on a file or a subset of the file and uses the local classifier to classify the file based on the feature extraction. In some embodiments, the edge device receives a data stream (e.g., a streaming file) and locally classifies the data stream based on an analysis of at least a subset of the data stream (e.g., one or more chunks of the streaming file). As one example, the edge device iteratively obtains chunks of the streaming data and predicts whether the streaming data (e.g., a streaming file) is malicious based at least in part on the chunks. The edge device may perform feature extraction on the chunks, query a local classifier based on results from the feature extraction (e.g., using one or more feature vectors obtained from the feature extraction), and obtain a prediction of the classification of the streaming data from the local classifier. As one example, in the case of security analysis, the prediction may correspond to the likelihood that the streaming data is malicious. The edge device may compare the prediction of the classification of the streaming data to a corresponding likelihood threshold for such classification (e.g., in the case of assessing whether the streaming data is malicious, a pre-defined maliciousness threshold). In response to comparing the prediction of the classification to the corresponding likelihood threshold (e.g., a threshold for a GDPR classification, a threshold for a PII classification, a threshold for a financial information classification, etc.), the edge device may consider the streaming data to correspond to the classification if the prediction of the classification exceeds the likelihood threshold, or conversely, consider the streaming data not to correspond to the classification if the prediction of the classification is less than (or equal to) the likelihood threshold.The edge device may then process the streaming data according to classification or other traffic, if applicable.

[0052] As an illustrative example in the case of security analysis performed on streaming data, if a prediction of whether the streaming data is malicious exceeds a likelihood threshold, the edge device may consider the streaming data to be malicious. Conversely, if a prediction of whether the streaming data is malicious is less than (or equal to) the likelihood threshold, the edge device may consider the streaming data to be non-malicious (e.g., benign). In response to determining (e.g., predicting) a classification of the streaming data, the edge device may implement / enforce applicable policies.

[0053] According to various embodiments, an edge device performs classification of streaming data (e.g., a streaming file) before the entire set of streaming data (e.g., the entire streaming file) is processed / received at the edge device. With reference to FIG. 1 , a data appliance 102 receives traffic data, such as streaming data. In response to receiving the traffic data, the data appliance 102 may perform classification of the traffic data. For example, the data appliance performs classification of the streaming file based at least in part on one or more chunks of the streaming file. As one example, a chunk may be a predetermined number of bytes of data (e.g., 1500 bytes of data). The data appliance 102 may sequentially analyze a set (or subset) of chunks of the streaming file (e.g., simultaneously with receipt of the chunks) and perform classification predictions on the streaming file before the entire streaming file is received / processed by the data appliance 102. In response to classifying the streaming file, the data appliance may implement a policy for processing the streaming data (e.g., chunk-by-chunk as each chunk is sequentially analyzed for classification). One example of enforcing a policy includes taking active action with respect to a streaming file in response to a particular classification of chunks of the streaming file (e.g., if the prediction that the file corresponds to a particular classification exceeds a predefined classification threshold, such as in response to the prediction indicating that the streaming data is malicious).

[0054] The data appliance 102 stores classifiers (e.g., machine learning models) used to locally classify traffic data, such as streaming files. The classifiers can be trained by a remote server, such as security platform 140, and provided to the data appliance 102 for inline classification. Additionally, the classifiers can be updated or retrained by the remote server, and the data appliance 102 is provided with the updated classifiers.

[0055] In some embodiments, the profile of traffic data / files received at an edge device, such as the data appliance 102, is nonlinear. Streaming files received at an edge device typically include header information, at least in the first chunk or packet of data. The header information skews other data included in the streaming file. For example, the header information corresponds to an offset, which shifts entity information within the streaming file. To ensure that classification of streaming files using chunks (e.g., classifying streaming files by chunk) is deterministic, various embodiments implement alignment of information included in chunks. For example, the data appliance 102 aligns information included in chunks to ensure that the data appliance 102 analyzes the same type of bytes (e.g., bytes containing entity information or non-header information, but not header information) (e.g., by using a classifier). In connection with performing the alignment of the information contained in the chunks, the data appliance 102 obtains a second data set from the first chunk (e.g., the last X bytes of the first chunk) and a first data set from the second chunk (e.g., the first Y bytes of the second chunk), considers the second data set from the first chunk and the first data set from the second chunk as data for a single specific chunk, and classifies the streaming file using the second data set from the first chunk and the first data set from the second chunk (e.g., using the single specific chunk). X and Y may be predefined positive integers. In some embodiments, the data appliance 102 stores a mapping of file types to X and Y values, and in response to starting to receive a streaming file, the data appliance 102 determines the file type for the streaming file and obtains the corresponding X and Y values.The data appliance 102 then uses the X and Y values ​​to align the chunk data within the set of chunks of the streaming file (eg, to take into account header information).

[0056] In connection with performing classification of the streaming file based on the chunks (e.g., chunk data), the data appliance 102 obtains a prediction for the classification of the streaming file. The prediction may correspond to the likelihood that the streaming file corresponds to a particular classification (e.g., the likelihood that the streaming file is malicious). In various embodiments, the data appliance 102 compares the classification prediction with a predefined classification threshold and, based on the result of the comparison, determines whether to regard the streaming file according to the predicted classification. For example, if the likelihood that the streaming file is malicious is greater than a predefined maliciousness threshold, the data appliance 102 considers the streaming file to be malicious and processes the traffic data (e.g., the streaming file) accordingly. According to various embodiments, once the data appliance 102 obtains a prediction for the streaming file based on the applicable chunk data, the data appliance 102 processes the streaming data according to the predicted classification. If the prediction of the chunk classification indicates (e.g., deemed to indicate based on the prediction meeting a predefined threshold) that the streaming file corresponds to a particular classification (e.g., a predefined classification such as malicious, PII, financial data, GDPR data, etc.), the data appliance 102 processes the remainder of the streaming file according to the classification. For example, in the case of security processing in the data appliance 102, if the first chunk has a predicted classification that meets a predefined maliciousness threshold, the data appliance 102 treats the streaming file as malicious (e.g., treats the current chunk and all future chunks of the streaming file as malicious).

[0057] In some embodiments, the data appliance 102 uses dynamic classification thresholds in connection with classifying traffic data using chunks. For example, chunks received earlier (e.g., at the beginning of the streaming data) may be classified using a first classification threshold, and chunks received later (e.g., at the end of the streaming file) may be classified using a second classification threshold. The first and second classification thresholds are different. In some implementations, the first classification threshold is higher than the second classification threshold. For example, the data appliance 102 lowers the classification threshold (e.g., the maliciousness threshold) as data in a streaming file is processed. Various classification thresholds or changes to the dynamic classification threshold may be set based on empirical testing of streaming file classifications. The data appliance 102 may store a mapping of the number of a particular chunk (or percentile of the total number of chunks) in a set of chunks to the applicable classification threshold used when classifying the particular chunk. As one example, a default malicious threshold is lower for the first chunk than for the jth chunk, where j is a positive integer greater than 1.

[0058] FIG. 2 is a block diagram of a system for classifying files, according to various embodiments. In some embodiments, system 200 is implemented, at least in part, by system 100 of FIG. 1. System 200 may be implemented by an inline security entity. In various embodiments, system 200 is implemented in conjunction with system 400 of FIG. 4, system 500 of FIG. 5, system 1300 of FIG. 13, and / or system 1400 of FIG. 14. In various embodiments, system 200 is implemented in conjunction with process 300 of FIG. 3, process 700 of FIG. 7, process 800 of FIG. 8, process 900 of FIG. 9, process 1000 of FIG. 10, and / or process 1100 of FIG. 11, process 1600 of FIG. 16, process 17 of FIG. 17, process 1800 of FIG. 18, and / or process 1900 of FIG. 19. System 200 may be implemented in one or more servers, security entities such as firewalls, and / or endpoints.

[0059] System 200 may be implemented by one or more devices, such as a server. System 200 may be implemented at various locations on a network. In some embodiments, system 200 implements data appliance 102 of system 100 of FIG. 1. As one example, system 200 is deployed as an edge device, such as a firewall or inline security entity, that performs inline security processing on traffic data (e.g., system 200 determines whether a file is malicious and processes the traffic data according to such malicious classification services). File classification may be implemented in conjunction with a locally stored classifier, such as a machine learning model. System 200 may receive the classifier from a server and store the classifier locally in its cache for use in inline file classification / processing.

[0060] According to various embodiments, in response to receiving traffic data to be analyzed (e.g., classified, such as to determine whether a file is malicious), system 200 performs feature extraction on the streaming file (e.g., performs feature extraction on specific chunks of data within the streaming file) and uses a classifier to classify the streaming file based on results (e.g., feature vectors) obtained from the feature extraction. System 200 processes the traffic data (e.g., the streaming file) according to the classification of the streaming file. System 200 iteratively performs feature extraction on chunks in a sequence of received / processed chunks from a set of chunks of the streaming file and classifies the streaming file using the corresponding results from the feature extraction. For example, system 200 receives chunks of the streaming file sequentially, and system 200 sequentially performs classification of the streaming file using specific chunks of data of the streaming data (e.g., the chunks are processed and classification is performed using the chunks in the order in which they were received).

[0061] In the illustrated example, system 200 implements one or more modules associated with classifying files, such as streaming files (e.g., predicting a classification), determining the likelihood that a file corresponds to a particular classification, and / or providing notification or indication of whether a file is malicious or not, or taking active action, in response to determining that the file classification matches a predetermined classification (e.g., in response to determining that the maliciousness prediction exceeds a predetermined maliciousness threshold). System 200 includes a communications interface 205, one or more processors 210, storage 215, and / or memory 220. The one or more processors 210 include one or more of a communications module 225, a chunk acquisition module 227, a chunk alignment module 229, a feature extraction module 231, a model training module 233, a prediction module 235, a notification module 237, and a security enforcement module 239.

[0062] In some embodiments, system 200 includes a communications module 225. System 200 uses communications module 225 to communicate with various nodes or endpoints (e.g., client terminals, firewalls, DNS resolvers, data appliances, other security entities, etc.) or user systems, such as administrator systems. For example, communications module 225 provides communications interface 205 with information to be communicated. As another example, communications interface 205 provides communications module 225 with information received by system 200. Communications module 225 is configured to receive a file to be analyzed from a network endpoint or node, such as a security entity (e.g., a firewall). Communications module 225 is configured to query a third-party service for information related to the file (e.g., a service that publishes information about the file, such as a third-party score or rating of the maliciousness of the file, a community-based score, rating, or reputation associated with the file, a blacklist for the file, and / or a whitelist for the file). Communications module 225 is configured to receive one or more settings or configurations from an administrator.Examples of the one or more settings or configurations include configuration of a process for determining whether a file is malicious, configuration related to a classifier or machine learning model for classifying files, predefined classification thresholds (e.g., predefined maliciousness thresholds, predefined financial data thresholds, etc.), settings related to header information about files or file types (e.g., header information / characteristics for various types of streaming files), a format or process according to which the combined feature vector is to be determined, a set of feature vectors to be provided to a classifier for classifying files (e.g., determining whether a file is malicious), information related to a whitelist of files (e.g., files that are not considered suspicious and for which traffic or attachments are allowed), and information related to a blacklist of files (e.g., files that are considered suspicious and for which traffic or attachments are restricted).

[0063] In some embodiments, system 200 includes chunk acquisition module 227. System 200 uses chunk acquisition module 227 to receive traffic data, such as streaming files. Chunk acquisition module 227 determines and / or acquires chunks that system 200 receives (e.g., in connection with monitoring network traffic). As one example, a chunk may be a predetermined number of bytes of data (e.g., 1500 bytes of data). Chunk acquisition module 227 may use a particular chunk definition (e.g., a predetermined number of bytes) for all file types, or chunk acquisition module 227 may use different chunk definitions based on file type. For example, system 200 stores a mapping of file types to chunk definitions (e.g., the number of bytes considered to be a chunk), and chunk acquisition module 227 queries the mapping of file types to chunk definitions to determine the chunk definition to use for a particular file type. The chunk acquisition module 227 acquires a set of chunks corresponding to a streaming file, and the chunks within the set of chunks may be received sequentially (e.g., in the order the chunks are arranged in the streaming file) or may be appropriately ordered by the chunk acquisition module 227 before processing. In response to receiving / processing the chunks, the chunk acquisition module 227 provides the chunks to the feature extraction module 231 for analysis and classification. In some embodiments, in contexts where the profile of the streaming data is non-linear and the streaming file being received by the system 200 includes header information in the first chunk or otherwise misaligned / unaligned chunks, the chunk acquisition module 227 provides the chunk data (e.g., one or more chunks of the streaming file) to the chunk alignment module 229 for chunk alignment before classification.

[0064] In some embodiments, system 200 includes chunk alignment module 229. System 200 uses chunk alignment module 229 to perform alignment of chunk data within a set of chunks of a streaming file. The chunk data is aligned to ensure that classification of various chunks is deterministic. Chunk alignment module 229 aligns chunk data to account for header information included in the first chunk or other misalignment of chunk data with respect to the chunk. As one example, a chunk may be a predetermined number of bytes of data (e.g., 1500 bytes of data). The default number of bytes may be configurable, such as by an administrator (e.g., the default number of bytes may be preset in a policy, such as a chunking policy or a file classification policy).

[0065] In some embodiments, the chunk alignment module 229 aligns chunk data of multiple chunks to ensure that the same type of data (e.g., non-header information) is analyzed during classification of the streaming file. The chunk alignment module 229 aligns chunk data of multiple chunks by considering a subset of data from each of two consecutive chunks as chunk data for a single chunk for which classification is to be performed. The chunk alignment module 229 may align chunk data of multiple chunks by obtaining a first predetermined number of bytes from a first chunk and a second predetermined number of bytes from a second chunk (e.g., the chunk immediately following the first chunk in the streaming file), and then consider the aggregate data of the first predetermined number of bytes and the second predetermined number of bytes as a single chunk to be used for classifying the streaming file. For example, if a chunk is set as 1500 bytes, and If the first chunk of a streaming file includes 500 bytes of header information, the chunk alignment module 229 uses the last 1000 bytes of the first chunk and the first 500 bytes of the second chunk and considers such data to correspond to a single chunk of data. The first and second predetermined number of bytes may be configurable. In some embodiments, the first predetermined number of bytes is determined based on the number of bytes included in the header information for the streaming file. By aligning the chunks, the system can analyze (e.g., run a machine learning model on) the same number of bytes in connection with classifying the streaming file based on the chunks.

[0066] In some embodiments, system 200 includes feature extraction module 231. System 200 uses feature extraction module 231 to perform feature extraction on particular chunks. For example, feature extraction module 231 performs feature extraction on chunks acquired by chunk acquisition module 227. As another example, such as when a streaming file has predefined header information, feature extraction module 231 performs feature extraction on aligned chunks (e.g., chunk data deemed to be chunks by chunk alignment module 229, e.g., based on information acquired from consecutive chunks).

[0067] In some embodiments, system 200 uses feature extraction module 231 to determine a set of feature vectors or combined feature vectors for use in connection with classifying a sample, such as determining whether a sample (e.g., a streaming file) is malicious (e.g., using a detection model). In some embodiments, the set of one or more feature vectors is determined based at least in part on information about the sample. For example, feature extraction module 231 determines (e.g., characterizes) a feature vector for one or more of: (i) a set of regular expression statements (e.g., pre-defined regular expression statements) and / or (ii) one or more properties or relationships determined based on algorithm-based feature extraction.

[0068] In some embodiments, system 200 (e.g., prediction module 235) uses the combined feature vector in connection with determining whether a sample is malicious or suspicious or to filter (e.g., remove) benign traffic. In some embodiments, system 200 (e.g., prediction module 235) uses the combined feature vector in connection with classifying streaming files (e.g., determining whether a file is malicious or another classification, such as GDPR data, financial data, or export-controlled data). Feature extraction module 231 may determine such a combined feature vector. The combined feature vector is determined at least in part based on a set of one or more feature vectors (e.g., based on a set of features for a model, such as a set of detection features in a model for determining whether a sample is malicious). For example, the combined feature vector is determined at least in part based on a set of feature vectors for a predefined set of regular expression statements and a set of feature vectors for properties or relationships determined based on algorithm-based feature extraction. The feature extraction module 231 determines a combined feature vector by concatenating a set of feature vectors for a predefined set of regular expression statements and / or a set of feature vectors for properties or relationships determined based on algorithm-based feature extraction. The feature extraction module 231 concatenates the set of feature vectors according to a predetermined process (e.g., a predetermined order, etc.).

[0069] In some embodiments, system 200 includes a model training module 233. System 200 uses model training module 233 to determine a model (e.g., a classifier) ​​for classifying streaming files. Model training module 233 may determine multiple models for classifying files along different vectors, such as classifying whether (i) the file is malicious, (ii) the file contains financial information, (iii) the file contains PII information, (iv) the file contains GDPR data, (v) the file contains export-controlled data, or (vi) the file contains another type of characteristic according to which a processing policy can be applied. Model training module 233 may determine relationships (e.g., features) between characteristics of a file (e.g., a streaming file) and a particular classification, such as the file's maliciousness (or the likelihood that the file is malicious). Examples of machine learning processes that may be implemented in connection with training a model include random forests, linear regression, support vector machines, naive Bayes, logistic regression, K-nearest neighbors, decision trees, gradient-boosted decision trees, K-means clustering, hierarchical clustering, density-based spatial clustering of noisy applications (DBSCAN) clustering, principal component analysis, etc. In some embodiments, the model training module 233 trains an XGBoost machine learning classifier. The input to the classifier (e.g., an XGBoost machine learning classifier model) is a combined feature vector or set of feature vectors, and based on the combined feature vector or set of feature vectors, the classifier model determines whether a corresponding .NET file is malicious or the likelihood that the .NET file is malicious.

[0070] In some embodiments, the models implemented by system 200 to classify streaming files are trained by a server. System 200 may use model training module 233 to obtain the models from the server. For example, model training module 233 may communicate with the server to determine whether a particular model is available or whether a particular model has been updated. Model training module 233 may query the server according to a preset frequency or according to a model training / update policy, which may be otherwise configurable, such as by an administrator.

[0071] In some embodiments, the system 200 includes a prediction module 235. The system 200 uses the prediction module 235 to predict the classification of a file. As one example, the prediction module 235 predicts whether a streaming file corresponds to a particular classification (e.g., malicious, PII, financial data, export-controlled data, GDPR data, etc.). Predicting the particular classification of a file may include predicting the likelihood that the file corresponds to the particular classification, comparing the predicted likelihood with a predefined classification threshold, and determining whether the file corresponds to the particular classification based on the comparison. For example, if the predicted likelihood exceeds the predefined classification threshold, the prediction module 235 considers the file to correspond to the particular classification. As another example, if the predicted likelihood does not exceed (e.g., is less than) the predefined classification, the prediction module 235 considers the file not to correspond to the particular classification.

[0072] The prediction module 235 uses a model, such as a machine learning model trained by the model training module 233 (or obtained from a server), in connection with determining whether a file corresponds to a particular classification (e.g., to predict the classification of the file, such as whether the file is malicious). For example, the prediction module 235 uses an XGBoost machine learning classifier model to analyze a combined feature vector obtained based on feature extraction of chunk data of a particular chunk to determine the classification of the streaming file. As another example, the prediction module 235 uses a convolutional neural network model to analyze features / characteristics (e.g., feature vectors) based on feature extraction of chunk data of the streaming file.

[0073] In some embodiments, the prediction module 235 iteratively classifies the streaming file based on the next chunk to be analyzed. For example, the prediction module 235 performs classification of the streaming file based on a single chunk (or a single aligned chunk) of the streaming file. The prediction module 235 may sequentially analyze chunks within the streaming file and determine a classification of the file for each chunk so analyzed. For example, the prediction module 235 determines a predicted classification for the file for each chunk within the streaming file, or until the prediction module 235 concludes that the predicted classification for a particular chunk exceeds a predetermined classification threshold (e.g., in which case the prediction module 235 considers the streaming file to correspond to the predicted classification). For each chunk of the streaming file, the system 200 may determine how to handle the streaming file (e.g., whether to allow transmission of the file, processing of the file, such as rendering the file, etc.).

[0074] In some embodiments, system 200 includes a notification module 237. System 200 uses notification module 237 to provide a classification indication for a file (e.g., to provide an indication that a sample streaming file is malicious). For example, notification module 237 obtains a file classification indication (or the likelihood that a sample corresponds to a particular classification) from prediction module 235 and provides the classification indication to one or more security entities and / or one or more endpoints.

[0075] In some embodiments, system 200 includes security enforcement module 239. System 200 uses security enforcement module 239 to enforce one or more security policies with respect to information such as network traffic, streaming files, etc. Security enforcement module 239 enforces one or more security policies based on the classification of a file. As one example, system 200 stores policies corresponding to different classifications, and these policies dictate how the file is handled. Examples of policies that security enforcement module 239 may enforce include a policy for handling malicious files, a policy for handling files containing financial information, a policy for handling files containing GDPR data, a policy for handling files containing export-controlled information, a policy for handling files containing PII, etc.

[0076] As one example, if system 200 is a security entity or firewall, system 200 includes security enforcement module 239. Firewalls typically deny or allow network transmissions based on a set of rules. These sets of rules are often referred to as policies (e.g., network policies, network security policies, security policies, etc.). For example, a firewall can filter inbound traffic by applying a set of rules or policies to prevent unwanted external traffic from reaching a protected device. A firewall can also filter outbound traffic by applying a set of rules or policies (e.g., allow, block, monitor, notify, or log, and / or other actions can be specified in a firewall rule or firewall policy, which can be triggered based on various criteria, as described herein). A firewall can also filter local network (e.g., intranet) traffic by applying a set of rules or policies. Other examples of policies include security policies that require scanning for threats in incoming (and / or outgoing) email attachments, website content, files exchanged through instant messaging programs, information obtained via other user interfaces such as web interfaces or interfaces to database systems (e.g., SQL interfaces), and / or other file transfers.

[0077] According to various embodiments, storage 215 includes one or more of file system data 260, model data 262, and / or prediction data 264. Storage 215 includes shared storage (e.g., a network storage system) and / or database data and / or user activity data.

[0078] In some embodiments, file system data 260 includes a database, such as one or more data sets (e.g., one or more data sets about files and / or file attributes, mappings of malicious indicators or other classifications to files or hashes, file signatures or other unique identifiers, mappings of benign file indicators to files or hashes, file signatures or other unique identifiers, etc.). File system data 260 includes data such as historical information about files (e.g., the maliciousness of the file), a whitelist of files that are considered safe (e.g., not suspicious), a blacklist of files that are considered suspicious or malicious (e.g., files that exceed a predefined / pre-set likelihood threshold to be considered malicious), information associated with suspicious or malicious files, etc. File system data 260 includes one or more policies, such as security policies for handling malicious files or other policies for handling other classifications.

[0079] Model data 262 includes information about one or more models (e.g., classifiers) used to classify files or predict the likelihood that a file will match a particular classification (e.g., the likelihood that a sample is malicious or suspicious). As one example, model data 262 includes a convolutional neural network model configured to classify streaming files. As another example, model data 262 stores a classifier (e.g., an XGBoost machine learning classifier model, such as a detection model, a prefilter model, or both) used in conjunction with a set of feature vectors or combined feature vectors. Model data 262 may include feature vectors generated for each of one or more of: (i) a set of regular expression statements; and / or (ii) algorithm-based features (e.g., features extracted using TF-IDF, such as for sample exploit traffic). In some embodiments, model data 262 includes a combined feature vector generated at least in part based on one or more feature vectors corresponding to each of one or more of: (i) a set of regular expression statements; and / or (ii) algorithm-based features (e.g., features extracted using TF-IDF, such as for sample exploit traffic).

[0080] Prediction data 264 includes information related to a determination of whether a sample analyzed by system 200 corresponds to a particular classification (e.g., a prediction of whether the sample is malicious). For example, prediction data 264 stores an indication that the sample is malicious, an indication that the sample is benign, etc. Information related to the determination may be obtained by notification module 237 and provided (e.g., communicated to an applicable security entity, endpoint, or other system). In some embodiments, prediction data 264 includes hashes or signatures of samples, such as samples analyzed by system 200 to determine whether such samples are malicious, or historical data sets previously evaluated for maliciousness, such as by a third party. Prediction data 264 may include a mapping of hash values ​​to indications of maliciousness (e.g., an indication of whether the corresponding sample is malicious or benign, etc.).

[0081] According to various embodiments, memory 220 includes running application data 270. Running application data 270 includes data obtained or used in connection with the execution of an application, such as an application performing a hash function, extracting information from a file, or analyzing the execution of a file in a sandbox. In embodiments, the applications include one or more applications that receive and / or perform queries or tasks, generate reports and / or configuration information responsive to the performed queries or tasks, and / or provide information responsive to the queries or tasks to a user. The other applications include any other suitable applications (e.g., index maintenance applications, communications applications, machine learning model applications, applications for detecting suspicious files, document creation applications, report creation applications, user interface applications, data analysis applications, anomaly detection applications, user authentication applications, security policy management / update applications, etc.).

[0082] Figure 3 is a block diagram of a method for classifying models. In some embodiments, process 300 is performed at least in part by system 100 of Figure 1 and / or system 200 of Figure 2. Process 300 may be performed by an inline security entity.

[0083] At 310, the samples are communicated. The samples may be communicated over a network or otherwise from one endpoint to another, etc.

[0084] At 320, the sample is obtained by a security entity, such as a firewall, configured to monitor traffic across a network or between two endpoints. In some embodiments, the firewall may be an application running on a client system and monitoring traffic to and from the client system.

[0085] At 330, the sample is analyzed using a machine learning model, such as an XGBoost model. Related art systems traditionally analyze a sample after it is fully received / processed. Related art systems obtain the entire sample, perform feature extraction on the sample (or a portion thereof, such as header information), and analyze the sample using a machine learning model. In some embodiments, a system (e.g., a firewall) performs feature extraction on a file, generates a feature vector, queries a model based at least in part on the feature vector, and obtains a result from the model. As one example, the result can be an indication of whether the file is malicious or non-malicious. As another example, the result can be an indication of the likelihood that the file is malicious or non-malicious. The system then compares the predicted likelihood to a pre-defined maliciousness threshold to determine whether to consider the file malicious.

[0086] In response to determining at 330 that the file is not malicious (e.g., benign), process 300 proceeds to 340, where the sample is treated as non-malicious traffic. For example, a firewall may allow the file to be transmitted or executed.

[0087] In response to determining that the file is malicious at 330, process 300 proceeds to 350, where the sample is treated as malicious. For example, a firewall may enforce one or more security policies with respect to the sample. As another example, the firewall may block transmission of the file.

[0088] Figure 4 illustrates a system for classifying streaming files based on a subset of chunks of the streaming files, according to various embodiments. In some embodiments, system 400 is implemented at least in part by system 100 of Figure 1 and / or system 200 of Figure 2. System 400 may be implemented by an inline security entity.

[0089] In some embodiments, system 400 is configured to provide a prediction of whether a file corresponds to a particular classification (e.g., whether the file is malicious) based on chunk data of a particular chunk of the file and before the entire file is received / processed. As one example, system 400 is deployed in a context in which streaming files are received. By predicting the classification of a streaming file, system 200 can provide low-latency prediction / processing decisions. For example, system 200 can decide to process the file according to a particular policy for classification based on a determination that the prediction obtained by using the chunks to predict the classification of the file meets the classification criteria / threshold (e.g., the prediction exceeds a predefined classification threshold, such as a predicted likelihood).

[0090] In some embodiments, system 400 is configured to provide predictions of whether a file corresponds to a particular classification locally at an edge device, such as a firewall, router, or other security entity. Due to memory and computational constraints of the edge device, system 400 is configured to generate predictions using relatively small models and using relatively small amounts of data (e.g., very little data is kept for classification).

[0091] 4, at 410, acquisition / processing of a sample begins, or at least a portion of a sample is acquired / processed. For example, system 400 begins receiving a streaming file corresponding to the sample. The streaming file is typically relatively large, and therefore, the data in the streaming file is streamed over a relatively long period of time.

[0092] At 420, the system acquires one or more chunks of the streaming file. In some embodiments, system 400 sequentially receives / processes chunks of the streaming file, such as chunk 421, chunk 422, chunk 423, chunk 424, chunk 425, etc. System 400 uses each particular chunk to determine a predicted classification of the streaming file and may determine, on a chunk-by-chunk basis, how to handle the streaming file (e.g., determine a policy to enforce regarding handling files of a particular classification). The system acquires chunk data from the chunk to be analyzed (e.g., each chunk of the streaming file, or each chunk until a particular classification is made). System 400 then provides the chunk data of a particular chunk to a convolutional neural network or other classifier to perform feature extraction and classification of the streaming file based on the chunk data.

[0093] At 430, the chunk data is input to a convolutional layer. In the example shown, chunk data of chunk 421 is input to convolutional layer 431, chunk 422 is input to convolutional layer 432, chunk 423 is input to convolutional layer 433, chunk 424 is input to convolutional layer 434, chunk 425 is input to convolutional layer 435, and chunk 426 is input to convolutional layer 436. The chunk data may be resized to an optimal size and then input to the corresponding convolutional layer.

[0094] The convolutional layer includes a filter or kernel used to filter the chunk data. For example, the kernel is placed on the chunk data. The extent to which the kernel is placed on or used to process a given chunk data is based on the kernel size (e.g., kernel dimension). In some embodiments, the performance (e.g., accuracy, speed, etc.) of streaming file classification can be adjusted based on the kernel size used by the convolutional layer. A relatively large kernel size generally has higher accuracy but requires a relatively large machine learning model, and the time to generate inferences / predictions is generally longer. Therefore, it may be preferable to select a kernel size that corresponds to a model size appropriate for a particular edge device and is fast enough to generate low-latency inferences, such as in accordance with quality-of-service policies or other configurations. In some embodiments, the kernel size is between 8 and 12. In some embodiments, the kernel size is 8. The inferences generated by the model are relatively quicker when the kernel size is 8 compared to larger kernels, such as a kernel size of 12, and the performance improvement is relatively small when comparing the implementation with a kernel size of 12 to a kernel size of 8. Thus, choosing a kernel size of 8 may be preferable / optimal when balancing the tradeoff between inference speed and accuracy.

[0095] In some embodiments, the kernel size used in the convolutional layer affects the number of characters used as lookbacks (e.g., the number of characters cached). The convolutional layer compares chunks of data one by one. For example, the convolutional layer uses different features for different portions of the chunks of data. The convolutional layer uses filters in connection with calculating matches between chunks of data and features (e.g., features corresponding to classifications for which a classifier is used to generate predictions).

[0096] Various embodiments implement a pooling mechanism in connection with analyzing chunks of data (e.g., generating predictions for a particular classification). Pooling is a mechanism for capturing a large amount of information and shrinking it, typically while preserving important information in the output. In the illustrated example, outputs from convolutional layers 431-436 are input to corresponding ones of max pooling modules 441-446, respectively. For example, at 440, the output from convolutional layer 431 is input to max pooling module 441. The max pooling module is configured to perform a max pooling operation on the information output by the applicable convolutional layer. A max pooling operation is a pooling operation that calculates the maximum (or largest value) of each feature map. As one example, a max pooling operation is a downsampling operation that results in downsampled feature maps that best emphasize features present in the data.

[0097] At 450, the output from the max pooling operation (eg, the output from max pooling modules 441-446) is used to generate a maximum layer.

[0098] At 460, the max layer is processed by a dense layer, which may include applying an activation function, such as a softmax operation, to the input data. The dense layer converts the feature maps output from the max layer into probability distributions.

[0099] At 470, the output from the dense layer is used to generate a prediction (e.g., inference). For example, the probability distribution output by the softmax operation is used to determine a prediction of the likelihood that the streaming file corresponds to a particular classification. The system may compare the likelihood prediction to a predefined classification threshold and determine a classification based on the results of the comparison. For example, the system may consider the streaming file to correspond to a particular classification if the prediction exceeds the corresponding predefined classification threshold.

[0100] 5 illustrates a system for classifying a streaming file based on a subset of chunks of the streaming file, according to various embodiments. In some embodiments, system 500 is implemented at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. System 500 may be implemented by an inline security entity.

[0101] At 510, acquiring / processing of the sample begins, or at least a portion of the sample is acquired / processed. For example, the system 500 begins receiving a streaming file corresponding to the sample.

[0102] At 520, the system obtains one or more chunks of the streaming file. In some embodiments, the system 500 sequentially receives / processes chunks of the streaming file, such as chunk 521, chunk 522, chunk 523, chunk 524, chunk 525, etc. The system 500 may use each particular chunk to determine a predicted classification of the streaming file and may then determine, on a chunk-by-chunk basis, how to handle the streaming file (e.g., determine a policy to enforce regarding handling files of a particular classification). The system obtains chunk data from the chunk to be analyzed (e.g., each chunk of the streaming file, or each chunk until a particular classification is reached). The system 500 then provides the chunk data of the particular chunk to a convolutional neural network or other classifier to perform feature extraction and classification of the streaming file based on the chunk data.

[0103] At 530, chunk data of the received chunks of the streaming file is provided to a classifier for feature extraction and classification inference / prediction of the streaming file. Streaming classification inference / prediction may include or correspond to a prediction of whether the streaming file corresponds to a particular classification. In some embodiments, system 500 sequentially provides multiple chunks of the streaming file to the classifier for sequential feature extraction and inference.

[0104] Performing feature extraction and generating a classification prediction includes, at 531, processing the corresponding chunk data using a convolutional layer (e.g., a convolutional neural network). The system obtains a feature map for the chunk based on processing the chunk data using the convolutional layer. In some embodiments, the convolutional layer uses a kernel having a kernel size between 8 and 12. In some embodiments, the kernel size is 8.

[0105] In response to processing the chunk of data using the convolutional layer, the system 500 provides the output from the convolutional layer to a pooling mechanism. At 532, the output from the convolutional layer (e.g., for a particular chunk processed by the convolutional layer) is input to the pooling mechanism, which then performs a max pooling operation.

[0106] At 533, the system 500 provides the output from the pooling mechanism to generate a maximum layer of features associated with the chunk. The maximum layer is cached at 534 in a cached maximum features list, which can be used in connection with feature comparisons for the next chunk at 535.

[0107] At 536, the max layer is processed by a dense layer, which may include applying an activation function, such as a softmax operation, to the input data. The dense layer converts the feature maps output from the max layer into probability distributions.

[0108] At 540, the output from the dense layer is used to generate a prediction (e.g., inference). For example, the probability distribution output by the softmax operation is used to determine a prediction of the likelihood that the streaming file corresponds to a particular classification. The system 500 may compare the likelihood prediction to a predefined classification threshold and determine a classification based on the results of the comparison. For example, the system may consider the streaming file to correspond to a particular classification if the prediction exceeds the corresponding predefined classification threshold.

[0109] In the illustrated example, system 500 successively generates predictions for consecutive chunks at 541, 542, 543, and 544. Predictions 541-544 may correspond to the likelihood that the streaming file corresponds to a particular classification. For example, in the case of a model for predicting whether a streaming file is malicious, the prediction output at 540 may correspond to the likelihood that the streaming file is malicious (e.g., based on an analysis of the corresponding chunks). In response to obtaining a prediction, system 500 may compare the prediction to a predefined classification threshold. If the predefined classification threshold is 0.95 (or 95%), system 500 determines that the streaming file does not correspond to a particular classification (e.g., is not malicious) because predictions 541-543 are less than the predefined classification threshold. Conversely, system 500 determines that prediction 544 indicates a streaming file that corresponds to a particular classification because prediction 544 is greater than the predefined classification threshold.

[0110] FIG. 6 illustrates a graph of file classification performance using a subset of chunks of a streaming file, according to various embodiments. In the example shown in FIG. 6, results 600 indicate that malicious files were mostly identified based on classification using earlier chunks. By implementing various embodiments to classify streaming files based on classification of specific chunks, the system is able to classify streaming files relatively early in the processing of the streaming file. Thus, systems according to various embodiments can identify that a streaming file corresponds to an applicable classification (e.g., malicious) early in the processing of the streaming file, thereby enabling the system to determine how to process the streaming file early (e.g., before the complete streaming file is received / processed), such as by enforcing applicable policies on files deemed to correspond to the applicable classification. For example, 50% of malicious files were identified by the 36th chunk of a streaming file, and 75% of malicious files were identified by the 50th chunk of a streaming file.

[0111] 7 is a flowchart of a method for classifying a streaming file prior to processing the entire streaming file, according to various embodiments. In some embodiments, process 700 is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. Process 700 may be performed by an inline security entity.

[0112] At 702, a sequence of integers corresponding to the bytes of the chunk being analyzed is obtained. At 704, a collection operation is performed to provide a lookup to an unsigned integer between 0 and 256 and extract the value of its corresponding dimension 16. At 706, an unsqueeze operation is performed to expand the dimension by 1. At 708, the system transposes the dimensions of the information to fit the information into an applicable CNN format (e.g., a 1D CNN format). For example, the transposition operation may include determining the product of the batch, channel, and length. At 710, the system performs a convolution operation. For example, the system performs a 1D convolution along the length of bytes using a predetermined kernel size. In some embodiments, the kernel size is between 8 and 12. In some embodiments, the kernel size is 8. At 712, a squeeze operation is performed to remove dimensions with a single value. At 714, an addition operation is performed to bias the output of the convolution. At 716, a rectified linear activation function (ReLU) is applied to perform a nonlinear activation operation. At 718, a global max pooling operation is performed. For example, the system obtains a single maximum value across all bytes (e.g., the length of the sequence). At 720, the system performs a squeeze operation to reduce the dimensionality of the maximum activation value. At 722, an external input tracks the maximum activation. For example, the system caches the maximum activation value. At 724, the system performs a max pooling operation to obtain the maximum output and maximum activation of the global max pooling output. At 726, the system stores the output from the max pooling operation in a cache. At 728, the system performs a matrix multiplication operation with the weights of the linear layer. At 730, the system adds a bias to the output of the matrix multiplication operation.At 732, the system performs a nonlinear activation operation (e.g., applies a nonlinear activation function). At 734, the system performs a matrix multiplication using the weights of the linear layer. At 736, the system adds a bias to the output of the matrix multiplication operation. At 738, the system performs a softmax operation to obtain class probabilities (e.g., to obtain a probability distribution). At 740, the class probabilities are passed as output.

[0113] 8 is a flowchart of a method for classifying a streaming file prior to processing the entire streaming file, according to various embodiments. In some embodiments, process 800 is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. Process 800 may be performed by an inline security entity.

[0114] At 805, a stream of a file is acquired. Acquiring a stream of a file (e.g., a streaming file) includes successively receiving chunks of the streaming file. For example, the streaming file is acquired by an edge device.

[0115] At 810, a machine learning model is used to process the set of chunks associated with the stream of files. The system performs feature extraction on the chunks and queries the model using the features (e.g., feature vector / map). In response to querying the model, the system obtains a prediction. The prediction may correspond to a probability / likelihood that the streaming file corresponds to a particular classification.

[0116] At 815, the file is classified. In some embodiments, the system classifies the file based on a prediction obtained from the model. For example, the system compares the prediction generated based on analyzing the chunks to a predefined classification threshold. If the prediction exceeds the predefined classification threshold, the system considers the streaming file to correspond to a particular classification. For example, in the case of a model used to detect malicious files, if the predicted likelihood of the streaming file exceeds the predefined maliciousness threshold, the system considers the streaming file to be malicious.

[0117] At 820, a determination is made as to whether process 800 is complete. In some embodiments, process 800 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 800 should be paused or stopped, etc. In response to a determination that process 800 is complete, process 800 terminates. In response to a determination that process 800 is not complete, process 800 returns to 805.

[0118] In some embodiments, the system may perform process 800 for each successive chunk of the streaming file until all chunks have been processed or until the system considers the streaming file to correspond to a particular classification (e.g., the classification for which the model is being developed).

[0119] 9 is a flowchart of a method for classifying a streaming file prior to processing the entire streaming file, according to various embodiments. In some embodiments, process 900 is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. Process 900 may be performed by an edge device, such as an inline security entity.

[0120] Process 900 may be implemented to determine whether a streaming file is malicious, for example, by analyzing each successive chunk (at least until the file is deemed malicious) and then classifying the file based on the analysis of the chunks.

[0121] A stream of the file is obtained at 905. In some embodiments, 905 corresponds to or is similar to 805 of process 800.

[0122] At 910, a machine learning model is used to process the set of chunks associated with the stream of files. In some embodiments, 910 corresponds to or is similar to 810 of process 800.

[0123] The files are classified at 915. In some embodiments, 915 corresponds to or is similar to 815 of process 800.

[0124] At 920, the system determines whether the file is malicious. The system determines whether the output of comparing the predictions obtained from the model beyond the predefined classification file indicates that the file is malicious.

[0125] In response to determining at 920 that the file is malicious, process 900 proceeds to 925, where one or more security policies are applied to the file. Conversely, in response to determining at 920 that the file is not malicious, process 900 proceeds to 930, where the file is treated as non-malicious traffic.

[0126] At 935, a determination is made as to whether process 900 is complete. In some embodiments, process 900 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 900 should be paused or stopped, etc. In response to a determination that process 900 is complete, process 900 terminates. In response to a determination that process 900 is not complete, process 900 returns to 905.

[0127] 10 is a flowchart of a method for training a classification model, according to various embodiments. In some embodiments, process 1000 is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. Process 1000 may be performed by an inline security entity.

[0128] Process 1000 is implemented to train a model to detect malicious files. In some embodiments, process 1000 is performed by a server that provides the trained model to an edge device for inline sample classification.

[0129] At 1005, information regarding a historical set of malicious samples is obtained. In one example, the system may collect information from a third-party service (e.g., VirusTotal). TM ) information about the set of historical malicious samples. As another example, the system obtains information about the set of historical malicious samples based on manual tagging by a human operator.

[0130] At 1010, information related to a set of historical benign samples is obtained. In one example, the system may collect data from a third-party service (e.g., VirusTotal). TM) to obtain information about the set of historical benign samples. As another example, the system obtains information related to the set of historical benign samples based on manual tagging by a human operator.

[0131] At 1015, one or more relationships between characteristics of the sample and the maliciousness of the sample are determined. In some embodiments, the system determines whether the streaming file is malicious or characteristics related to the likelihood that the streaming file is malicious. The characteristics may be determined based on a malicious feature extraction process performed on the sample.

[0132] In some embodiments, the features may be determined with respect to a set of regular expression statements (e.g., predefined regular expression statements) and / or with respect to the use of algorithm-based feature extraction (e.g., TF-IDF, etc.).

[0133] In some embodiments, the system divides each sample into chunks. In one example, a chunk corresponds to a predetermined number of bytes. In response to obtaining a chunk from a historical sample, the system performs feature extraction on information contained in the chunk of the sample.

[0134] At 1020, a model is trained to determine whether a file is malicious. The model is a machine learning model that is trained using a machine learning process. In some embodiments, the model is trained using a convolutional neural network. Various other machine learning processes may be implemented. Examples of other machine learning processes that may be performed in connection with training the model include random forests, linear regression, support vector machines, naive Bayes, logistic regression, K-nearest neighbors, decision trees, gradient-boosted decision trees, K-means clustering, hierarchical clustering, density-based spatial clustering of applications with noise (DBSCAN) clustering, principal component analysis, etc.

[0135] At 1025, the model is deployed. In some embodiments, deploying the model includes storing the model in a models dataset for use in connection with analyzing the samples to classify them (e.g., determining whether the samples are malicious, if the model is a detection model that detects malicious samples). In some embodiments, deploying the model includes storing the model in a models dataset for use in connection with analyzing the samples to determine whether the samples are malicious or suspicious (e.g., if the model is a pre-filter model that pre-filters network traffic based on the detection of a malicious or suspicious sample). Deploying the model may include providing the model (or a location where the model can be invoked) to an edge device, such as a security entity.

[0136] At 1030, a determination is made as to whether process 1000 is complete. In some embodiments, process 1000 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 1000 should be paused or stopped, etc. In response to a determination that process 1000 is not complete, process 1000 returns to 1005.

[0137] 11 is a diagram of a set of chunks associated with a streaming file, according to various embodiments. In some embodiments, the ordering and / or classification of the set of chunks is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. The classification of the chunks may be performed by an inline security entity.

[0138] In the illustrated example, a streaming file includes a set of chunks, such as chunk 1105, chunk 1110, chunk 1115, and chunk 1120. Chunks 1105-1120 do not necessarily contain the same type of information. For example, the profile of a received file may be nonlinear. The file may include header information, and the header information may be included in the first chunk. However, because the model is deterministic for classifying files, the analyzed chunks contain the same type of information. Various embodiments address issues that arise for nonlinear file profiles by aligning chunk data within the chunks.

[0139] In some embodiments, aligning the chunk data includes considering a first subset of the chunk data of a particular chunk and a second subset of the chunk data of a different chunk as chunk data for a single chunk. For example, the system obtains a first predetermined number of bytes from a particular chunk and a second predetermined number of bytes from a consecutive chunk, and collectively considers such information as chunk data for a single chunk. In response to aligning the chunk data, the system classifies the streaming file based on analyzing various chunks within the streaming file (or until the system determines that the streaming file corresponds to a particular classification).

[0140] As shown in FIG. 11 , chunk 1105 includes header information 1130 and payload data 1140-1. Header information 1130 may be a predefined number of bytes based on the type of file. To align the chunk data (e.g., to determine chunks containing only payload data for analysis to generate classification predictions), the system obtains payload data 1140-1 from the first chunk and payload data 1140-2 from the second chunk. The system considers the chunk data for chunk 1105 to include payload data 1140-1 and payload data 1140-2. Obtaining payload data 1140-1 from the first chunk may include obtaining a predetermined number of bytes from a back of the chunk. As one example, the system may obtain the last X bytes from a particular chunk and then obtain the first Y bytes from a consecutive chunk. In response to obtaining chunk data for a particular chunk (e.g., chunk data for chunk 1105), the system queries a model based on the chunk data to obtain a prediction of whether the streaming file corresponds to a particular classification (e.g., whether the streaming file is malicious, etc.).

[0141] Similarly, the system considers (i) the payload 1150-1 of the second chunk and the payload 1150-2 of the third chunk as chunk data for chunk 1110 (e.g., a single chunk for processing to generate a prediction), and (ii) the payload 1160-1 of the third chunk and the payload 1160-2 included in the fourth chunk as chunk data for chunk 1115. Payload 1170-1 may be combined with a subset of the data included in subsequent chunks.

[0142] In response to aligning the chunks, the system uses the corresponding chunk data to make a prediction as to whether the streaming file corresponds to a particular classification.

[0143] 12 is a flowchart of a method for detecting malicious files, according to various embodiments. In some embodiments, the sorting and / or classification of the set of chunks is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. The classification of the chunks may be performed by an inline security entity.

[0144] Bytes are input to the chunk alignment mechanism at 1205. In some embodiments, the input bytes are bytes converted to unsigned integer values ​​between 0 and 255.

[0145] At 1210, a system (e.g., a chunk alignment mechanism) obtains bytes from chunk data of a streaming file and processes the set of chunks to align the chunk data. For example, the system takes a variable-length chunk size and divides the chunk data based on an offset from the first byte.

[0146] Referring to 1230, chunk alignment includes obtaining a subset of bytes from a first chunk and a subset of bytes from a second chunk. The subset of bytes obtained from the first chunk and / or the second chunk may be predefined, such as in a chunk alignment policy or based on the type of file being processed. In the illustrated example, a chunk includes 1500 bytes. In the illustrated example, 1000 bytes of the first chunk correspond to header information, which may be discarded (or ignored for generating a prediction of the file classification). For example, subset 1232 of bytes from the first chunk corresponds to payload data. To ensure that the same amount of payload data is analyzed to generate the prediction, the system obtains subsets of bytes from consecutive chunks. Thus, the system considers subset 1232 of bytes from the first chunk and subset 1236 of bytes from the second chunk 1234 as chunk data for a single chunk to be analyzed to generate the prediction. As shown, the subset of bytes obtained from the first chunk is 500 bytes, and because the predetermined chunk size is 1500 bytes, the subset of bytes 1236 contains 1000 bytes. A second subset of bytes 1238 of the second chunk 1234 is then used along with a subset of bytes from a consecutive chunk (e.g., the third chunk).

[0147] If the expected chunk size is 1500 bytes and the first chunk is also 1500 bytes (of payload data), the offset is 0, all packets are aligned, and alignment of chunk data is not required. Conversely, if the expected chunk size is 1500 bytes and the first chunk includes an offset (e.g., header information that causes an offset), the offset is used to split subsequent chunks to obtain information from the aligned chunks. The offset may be predefined based on the file type or may be determined by the system when the streaming file is received. In the illustrated example, the first chunk is 500 bytes (of payload data), therefore the offset is 1000 bytes, and the second chunk 1234 is then split into a first subset of bytes 1236 and a second subset of bytes 1238. The first subset of bytes 1236 and the second subset of bytes 1238 may be used in different chunks for chunk alignment and prediction generation.

[0148] Returning to 1210, the system (e.g., a chunk alignment mechanism, such as chunk alignment module 229 of system 200) obtains 1215 extracts of bytes of size k−1, where k is the maximum kernel size used in the model (e.g., a convolutional neural network). The extracts of the set of k−1 bytes may be the previous k−1 bytes, such as from a previous chunk. The system (e.g., chunk alignment mechanism) obtains maximum activation values ​​1220 for various chunks. For example, the system tracks the maximum activation of the aligned chunks (e.g., every 1500 bytes in the example above).

[0149] According to various embodiments, the system (e.g., chunk alignment mechanism) implements a predefined algorithm to align chunk data. The algorithm includes: [Table 1]

[0150] Figure 13 is a block diagram of a system for classifying streaming files based on chunk data, according to various embodiments. In some embodiments, system 1300 may be implemented at least in part by system 100 of Figure 1 and / or system 200 of Figure 2. Chunk classification may be performed by an inline security entity.

[0151] The system 1300 classifies files based on performing feature extraction on the chunk data and classification. The feature extraction includes processing the chunk data using a convolutional neural network 1305 and processing output from the convolutional neural network 1305 using a global maximum layer 1310. In the illustrated example, the convolutional neural network 1305 is a one-dimensional convolutional neural network with a kernel size of 12. However, various other kernel sizes may be implemented. In some embodiments, the kernel size is 8. In response to passing the chunk data through the convolutional neural network 1305, the system provides the output from the convolutional neural network 1305 to the global maximum layer 1310 to obtain a maximum activation (e.g., a maximum activation value). For example, the global maximum layer 1310 performs a max-pooling operation on the output from the convolutional neural network 1305.

[0152] In response to performing feature extraction, the system performs classification on the feature vector / map to obtain a prediction of whether the streaming file corresponds to a particular classification. Classifying the streaming file based on a particular chunk of data in the streaming file includes processing the output from the global max layer 1310 using a dense layer and softmax module 1315.

[0153] 14 is a block diagram of a classification of a set of chunks captured within a stream of a file, according to various embodiments. In some embodiments, the classification of the set of chunks is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. The classification of the chunks may be performed by an inline security entity.

[0154] System 1400 performs classification of a streaming file based on chunk data for a particular chunk. System 1400 acquires a streaming file based on receiving successive chunks. A first chunk includes header information 1405 and payload information 1410-1, a second chunk includes payload information 1410-2 (e.g., a first subset of bytes of the second chunk) and payload information 1415-1 (e.g., a second subset of bytes of the second chunk), a third chunk includes payload information 1415-2 (e.g., a first subset of bytes of the third chunk) and payload information 1420-1 (e.g., a second subset of bytes of the third chunk), and a fourth chunk includes payload information 1420-2 (e.g., a first subset of bytes of the fourth chunk) and payload information 1425-1 (e.g., a second subset of bytes of the fourth chunk). The first chunk contains header information 1405 , so the payload data of the chunk of the streaming file has an offset equal to the number of bytes in the header information 1405 .

[0155] The system 1400 sequentially applies the convolutional neural network model to successive chunks. For example, at 1440, the system 1400 passes the first chunk of payload information 1410-1 through the convolutional neural network, at 1442, the system 1400 passes payload information 1410-2 and 1415-1 through the convolutional neural network, at 1444, the system 1400 passes payload information 1415-2 and 1420-1 through the convolutional neural network, and at 1446, the system 1400 passes payload information 1420-2 and 1425-1 through the convolutional neural network.

[0156] In response to passing chunks (e.g., payload information) to the convolutional neural network, system 1400 passes outputs from the convolutional neural network through pooling layers. For example, at 1450, system 1400 passes outputs from 1440 through a global max layer, at 1452, system 1400 passes outputs from 1442 through a global max layer, at 1454, system 1400 passes outputs from 1444 through a global max layer, and at 1456, system 1400 passes outputs from 1446 through a global max layer. As one example, the global max layer performs a max pooling operation on the outputs from the convolutional layers.

[0157] At 1460, the system 1400 passes the output (e.g., maximum activation values) from the global maximum layer for payload information 1410-1 and payload information 1410-2 through a dense layer and softmax operation for the first virtual chunk (e.g., chunks aligned with payload information from the first received chunk and the second received chunk). Similarly, at 1465, the system 1400 passes the output (e.g., maximum activation values) from the global maximum layer for payload 1415-1 and payload 1452-2 through a dense layer and softmax operation for the second virtual chunk.

[0158] At 1470, the system 1400 uses the output from the dense layer and the softmax operation for the first virtual chunk to determine a classification of the streaming file based on the first virtual chunk. For example, the system 1400 generates a prediction of the likelihood that the streaming file corresponds to a particular classification. Similarly, at 1475, the system 1400 uses the output from the dense layer and the softmax operation for the second virtual chunk to determine a classification of the streaming file based on the second virtual chunk.

[0159] System 1400 uses successive chunk data (e.g., successive aligned chunks or virtual chunks) to successively classify whether a streaming file corresponds to a particular classification and generate a prediction. System 1400 may perform successive classification and generation of a prediction until the earlier of (i) all chunks in the streaming file have been processed for classification, and (ii) system 1400 determines that the streaming file corresponds to a particular classification based on the prediction (e.g., system 1400 determines that the prediction exceeds a predetermined classification threshold).

[0160] 15 is a block diagram of a classification of a set of chunks captured within a stream of a file, according to various embodiments. In some embodiments, the classification of the set of chunks is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. The classification of the chunks may be performed by an inline security entity.

[0161] The system 1500 classifies streaming files based on analysis of successive chunks. For example, when the system 1500 receives a chunk of a streaming file, the system 1500 processes the corresponding chunk data and generates a successive prediction of whether the streaming file is malicious or not.

[0162] In the illustrated example, the first chunk includes header information 1505 (or other offset) and payload information 1510-1, the second chunk includes payload information 1510-2 and 1515-1, the third chunk includes payload information 1515-2 and 1520-1, and the fourth chunk includes payload information 1520-2 and 1525-1.

[0163] The system 1500 performs feature extraction 1530-1 on payload information 1510-1 obtained from the first chunk, and performs feature extraction 1530-2 on payload information 1510-2 obtained from the second chunk. In response to performing feature extractions 1530-1 and 1530-2, the system 1500 performs classification 1535 of the streaming file (e.g., a prediction based on analysis of the first virtual chunk or payload information 1510-1 and 1510-2). For example, the system 1500 generates a prediction of whether the streaming file corresponds to a particular classification.

[0164] The system 1500 performs feature extraction 1540-1 on payload information 1515-1 obtained from the first chunk, and performs feature extraction 1540-2 on payload information 1515-2 obtained from the second chunk. In response to performing feature extractions 1540-1 and 1540-2, the system 1500 performs classification 1545 of the streaming file (e.g., a prediction based on analysis of the first virtual chunk or payload information 1515-1 and 1515-2). For example, the system 1500 generates a prediction of whether the streaming file corresponds to a particular classification.

[0165] The system 1500 performs feature extraction 1550-1 on payload information 1520-1 obtained from the first chunk, and performs feature extraction 1550-2 on payload information 1520-2 obtained from the second chunk. In response to performing feature extractions 1550-1 and 1550-2, the system 1500 performs classification 1555 of the streaming file (e.g., a prediction based on analysis of the first virtual chunk or payload information 1520-1 and 1520-2). For example, the system 1500 generates a prediction of whether the streaming file corresponds to a particular classification.

[0166] In some embodiments, system 1500 performs classification 1545 only if system 1500 does not deem the streaming file to correspond to a particular classification based on classification 1535. Similarly, system 1500 may perform classification 1555 only if system 1500 does not deem the streaming file to correspond to a particular classification based on classification 1545.

[0167] Figure 16 is a flowchart of a method for classifying a stream of files, according to various embodiments. In some embodiments, process 1600 is performed at least in part by system 100 of Figure 1 and / or system 200 of Figure 2. Process 1600 may be performed by an inline security entity.

[0168] At 1605, a stream of a file is acquired. Acquiring a stream of a file (e.g., a streaming file) includes successively receiving chunks of the streaming file. For example, the streaming file is acquired by an edge device.

[0169] In some embodiments, in response to receiving the first chunk of data of a streaming file, the system determines whether chunk alignment should be performed. For example, the system determines whether the first chunk includes an offset (e.g., header information). The system may determine that the first chunk includes an offset based on an analysis of the chunk, such as based on the file type.

[0170] At 1610, the system aligns a predetermined amount of data within chunks associated with the stream of files. In some embodiments, in response to determining an offset associated with the streaming file (e.g., the range to which payload information is offset within a chunk), the system performs chunk alignment to take the offset into account. For example, the system determines virtual chunks (also referred to herein as aligned chunks) and then sequentially performs classification of the streaming file based on consecutive virtual chunks.

[0171] In some embodiments, a virtual chunk contains a subset of bytes from a particular chunk and a subset of bytes from another chunk, such as a contiguous chunk, where the subset of bytes from a particular chunk can be a predetermined number of bytes at the end of the first chunk, and the subset of bytes in the contiguous chunk can be a predetermined number of bytes at the beginning of the contiguous chunk.

[0172] The aligned chunks are processed using the machine learning model at 1615. In some embodiments, the system performs feature extraction on the aligned chunks and, in response to performing the feature extraction, performs classification of the aligned chunks.

[0173] At 1620, the file is classified. In some embodiments, the file is classified by chunk. For example, the system classifies the file based on processing of a particular chunk by a machine learning model. The system may classify the file sequentially as the chunks are processed. In some embodiments, if the system determines that the predicted classification (based on a particular chunk) exceeds a pre-defined classification threshold, the system stops processing chunks of the streaming file.

[0174] At 1625, a determination is made as to whether process 1600 is complete. In some embodiments, process 1600 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 1600 should be paused or stopped, etc. In response to a determination that process 1600 is complete, process 1600 terminates. In response to a determination that process 1600 is not complete, process 1600 returns to 1605.

[0175] In response to classifying the streaming file with each contiguous chunk, the system may determine how the streaming file (e.g., a particular chunk) should be processed, such as whether a policy should be enforced with respect to the streaming file.

[0176] Figure 17 is a flowchart of a method for detecting malicious files, according to various embodiments. In some embodiments, process 1700 is performed at least in part by system 100 of Figure 1 and / or system 200 of Figure 2. Process 1700 may be performed by an inline security entity.

[0177] Process 1700 may be implemented to determine whether a streaming file is malicious. For example, process 1700 may analyze each successive chunk (at least until the file is deemed malicious) and classify the file based on the analysis of the chunks.

[0178] A stream of the file is obtained at 1705. In some embodiments, 1705 corresponds to or is similar to 1605 of process 1600.

[0179] In 1710, the system aligns a predetermined amount of data in chunks associated with a stream of a file. In some embodiments, 1710 corresponds to or is similar to 1610 of process 1600.

[0180] The aligned chunks are processed using the machine learning model at 1715. In some embodiments, 1715 corresponds to or is similar to 1615 of process 1600.

[0181] At 1720, the file is sorted. In some embodiments, the file is sorted by chunk. In some embodiments, 1720 corresponds to or is similar to 1620 of process 1600.

[0182] At 1725, the system determines whether the file is malicious. The system determines whether the output of comparing the prediction obtained from the model exceeds the predefined classification file, which indicates that the file is malicious.

[0183] In response to determining that the file is malicious at 1725, process 1700 proceeds to 1730, where one or more security policies are applied with respect to the file. The system may process malicious traffic / information based at least in part on one or more policies, such as one or more security policies.

[0184] According to various embodiments, processing malicious sample traffic / information may include performing active actions. The active actions may be performed according to (e.g., based at least in part on) one or more security policies. As one example, one or more security policies may be pre-configured by a network administrator, a customer (e.g., an organization / company) for a service that provides detection of malicious input strings, files, etc. Examples of active actions that may be performed include quarantining the sample (e.g., quarantining the sample), deleting the sample (e.g., deleting chunk data for one or more chunks), alerting a user that a malicious sample was detected, providing a prompt to a user when a device attempts to open or execute the sample, blocking transmission of the sample, updating a blacklist of malicious input strings (e.g., mapping a sample's hash to an indication that the sample is malicious), etc.

[0185] In response to determining at 1725 that the traffic does not contain a malicious sample, process 1700 proceeds to 1735, where the sample (e.g., a streaming file) is treated as a non-malicious sample (e.g., non-malicious traffic / information). For example, the system may process the non-malicious sample according to normal operations (e.g., allowing the file to be sent / communicated, etc.).

[0186] At 1740, a determination is made as to whether process 1700 is complete. In some embodiments, process 1700 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 1700 should be paused or stopped, etc. In response to a determination that process 1700 is complete, process 1700 terminates. In response to a determination that process 1700 is not complete, process 1700 returns to 1705.

[0187] Figure 18 is a flowchart of a method for detecting malicious files, according to various embodiments. In some embodiments, process 1800 is performed at least in part by system 100 of Figure 1 and / or system 200 of Figure 2. Process 1800 may be performed by an inline security entity.

[0188] Process 1800 may be implemented to determine whether a streaming file is malicious. For example, process 1800 may analyze each successive chunk (at least until the file is deemed malicious) and classify the file based on the analysis of the chunks.

[0189] Process 1800 illustrates one example where the system uses each successive analysis / classification of chunks within a streaming file to determine how to treat the streaming file.

[0190] A stream of the file is obtained at 1805. In some embodiments, 1805 corresponds to or is similar to 1605 of process 1600.

[0191] In 1810, n is set equal to 1. n is a positive integer that is used as a counter during processing of chunks of the stream.

[0192] At 1815, a predetermined subset of the nth chunk is obtained.

[0193] At 1820, a predetermined subset of the (n+1)th chunk is obtained.

[0194] At 1825, feature extraction is performed on a predetermined subset of the nth chunk and a predetermined subset of the (n+1)th chunk.

[0195] At 1830, a model is queried based on the feature extraction.

[0196] A prediction is obtained from the model at 1835. The model may provide a predicted classification or the likelihood that a file corresponds to a particular classification.

[0197] At 1840, the system determines whether the prediction is greater than the maliciousness threshold.

[0198] In response to determining at 1840 that the prediction is greater than the maliciousness threshold, process 1800 proceeds to 1855, where one or more security policies are applied with respect to the file (or at least any future received / processed chunks of the file). As one example, if the prediction is greater than the maliciousness threshold, the system considers the file to be malicious.

[0199] Conversely, in response to determining at 1840 that the prediction is not greater than the maliciousness threshold, process 1800 proceeds to 1845, where the system determines whether the file is complete. The system may determine whether the file is complete based on determining whether the most recent chunk processed (e.g., the (n+1)th chunk) was the last chunk of the streaming file.

[0200] In response to determining at 1845 that the file is complete, process 1800 proceeds to 1860. Conversely, in response to determining at 1845 that the file is not complete, process 1800 proceeds to 1850, where n is incremented (e.g., n=n+1). Process 1800 then returns to 1815, and process 1800 iterates through 1815-1840 until processing of the file is complete (e.g., a file classification is predicted for each chunk) or the system classifies the file as malicious based on analyzing the chunk data for a particular chunk.

[0201] At 1860, a determination is made as to whether process 1800 is complete. In some embodiments, process 1800 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 1800 should be paused or stopped, etc. In response to a determination that process 1800 is complete, process 1800 terminates. In response to a determination that process 1800 is not complete, process 1800 returns to 1805.

[0202] 19 is a flowchart of a method for detecting malicious files, according to various embodiments. In some embodiments, process 1900 is performed at least in part by system 100 of FIG. 1 and / or system 200 of FIG. 2. Process 1900 may be performed by an inline security entity.

[0203] Process 1900 may be implemented to determine whether a streaming file is malicious. For example, process 1900 may analyze each successive chunk (at least until the file is deemed malicious) and classify the file based on the analysis of the chunks. While process 1900 illustrates one example for classifying a file as malicious or not, various other embodiments may be implemented to determine other classifications of streaming files.

[0204] Process 1900 illustrates one example in which the system uses each successive analysis / classification of chunks within a streaming file to determine how to treat the streaming file.

[0205] A stream of the file is obtained at 1905. In some embodiments, 1905 corresponds to or is similar to 1805 of process 1800.

[0206] In 1910, n is set equal to 1. In some embodiments, 1910 corresponds to or is similar to 1805 of process 1800.

[0207] At 1915, the last X bytes of the nth chunk are obtained. X is a positive integer. In some embodiments, X is predefined. In some embodiments, X is determined based on the number of bytes contained in the header information of the streaming file.

[0208] At 1920, the first Y bytes of the (n+1)th chunk are obtained. Y is a positive integer. In some embodiments, Y is predefined. In some embodiments, Y is determined based on the number of bytes contained in the header information of the streaming file.

[0209] At 1925, feature extraction is performed on the last X bytes of the nth chunk and the first Y bytes of the (n+1)th chunk.

[0210] Based on the feature extraction, a model is queried at 1930. In some embodiments, 1930 corresponds to or is similar to 1830 of process 1800.

[0211] A prediction is obtained from the model at 1935. In some embodiments, 1935 corresponds to or is similar to 1835 of process 1800.

[0212] In 1940, the system determines whether the prediction is greater than the maliciousness threshold. In some embodiments, 1940 corresponds to or is similar to 1840 of process 1800.

[0213] In response to determining at 1940 that the prediction is greater than the maliciousness threshold, process 1900 proceeds to 1955, where one or more security policies are applied with respect to the file (or at least any future received / processed chunks of the file). As one example, if the prediction is greater than the maliciousness threshold, the system considers the file to be malicious.

[0214] Conversely, in response to determining at 1940 that the prediction is not greater than the maliciousness threshold, process 1900 proceeds to 1945, where the system determines whether the file is complete. The system may determine whether the file is complete based on determining whether the most recently processed chunk (e.g., the (n+1)th chunk) was the last chunk of the streaming file.

[0215] In response to determining at 1945 that the file is complete, process 1900 proceeds to 1960. Conversely, in response to determining at 1945 that the file is not complete, process 1900 proceeds to 1950, where n is incremented (e.g., n=n+1). Process 1900 then returns to 1915, and process 1900 iterates through 1915-1940 until processing of the file is complete (e.g., a classification of the file is predicted for each chunk) or the system classifies the file as malicious based on analyzing the chunk data for a particular chunk.

[0216] At 1960, a determination is made as to whether process 1900 is complete. In some embodiments, process 1900 is determined to be complete in response to a determination that no more samples should be analyzed (e.g., no more predictions of samples are required), no more traffic should be analyzed, an administrator indicates that process 1900 should be paused or stopped, etc. In response to a determination that process 1900 is complete, process 1900 terminates. In response to a determination that process 1900 is not complete, process 1900 returns to 1905.

[0217] Various examples of the embodiments described herein are described with reference to flowcharts. Although the examples may include particular steps performed in a particular order, according to various embodiments, various steps may be performed in different orders and / or various steps may be combined into a single step or in parallel.

[0218] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not limiting.

Claims

1. 1. A system for performing classification at an edge device, comprising: The system includes one or more processors and a memory; The one or more processors: obtaining a stream of a file at the edge device; using a machine learning model to process a set of chunks associated with the stream of the file; and classifying the file at the edge device before processing the file in its entirety; It is structured as follows: The memory includes: coupled to the one or more processors; and providing instructions to said one or more processors; It is configured as follows: system.

2. The edge device is a network device. The system of claim 1 .

3. the edge device is an inline security entity; The system of claim 1 .

4. the machine learning model is configured to classify the file as malicious or not; The system of claim 1 .

5. the machine learning model is configured to classify whether the file is copyrighted material; The system of claim 1 .

6. the machine learning model is configured to classify the file as being health data or financial data; The system of claim 1 .

7. If the prediction obtained from the machine learning model exceeds a predefined maliciousness threshold, the file is determined to be malicious. The system of claim 1 .

8. the file is determined to be malicious after the nth chunk is processed using the machine learning model; n corresponds to a positive integer less than the total number of chunks in the file; The system of claim 7.

9. the predetermined maliciousness threshold is constant for each chunk in the file; The system of claim 7.

10. the predetermined maliciousness threshold is dynamic across classifications of chunks within the file; The system of claim 7.

11. the predetermined maliciousness threshold is lower for the first chunk than for the jth chunk; j is a positive integer greater than 1, The system of claim 10.

12. In response to determining that the file is malicious, active action is taken against the malicious file. The system of claim 7.

13. the active action includes dropping or blocking remaining chunks associated with the file; The system of claim 12.

14. Each chunk corresponds to m bytes, m is a positive integer, The system of claim 1 .

15. The machine learning model is trained using a deep learning process. The system of claim 1 .

16. The deep learning process includes a convolutional neural network. The system of claim 15.

17. the machine learning model is trained at least in part based on a recurrent neural network; and a max pooling operation is performed to maintain state information over at least a subset of the chunks associated with the file; The system of claim 15.

18. the machine learning model is trained on the entire file; The system of claim 1 .

19. 1. A method for performing classification at an edge device, comprising: obtaining a stream of a file at the edge device; processing a set of chunks associated with the stream of the file using a machine learning model; classifying the file at the edge device before processing the file in its entirety; A method comprising:

20. 1. A computer program for performing classification at an edge device, the computer program comprising a plurality of instructions stored on a non-transitory computer-readable storage medium, the computer program comprising: The instructions, when executed, cause one or more processors to: obtaining a stream of a file at the edge device; processing a set of chunks associated with the stream of the file using a machine learning model; classifying the file at the edge device before processing the file in its entirety; To implement Computer program.

21. 1. A system for performing classification at an edge device, comprising: The system includes one or more processors and a memory; The one or more processors: obtaining a stream of a file at the edge device; Aligning a predetermined amount of data within a chunk associated with the stream of the file; using a machine learning model to process the plurality of aligned chunks associated with the stream of the file; classifying, at the edge device, the file based at least in part on the classification of the plurality of aligned chunks. It is structured as follows: The memory includes: coupled to the one or more processors; and providing instructions to said one or more processors; It is configured as follows: system.

22. The edge device is a network device.

22. The system of claim 21.

23. the edge device is an inline security entity; 22. The system of claim 21.

24. the machine learning model is configured to classify the file as malicious or not; 22. The system of claim 21.

25. 22. The system of claim 21, wherein the file is classified using the machine learning model before the entire file is processed.

26. the first chunk includes overhead associated with the file; 22. The system of claim 21.

27. Aligning a predetermined amount of data within a chunk associated with the stream of the file includes: determining a first file segment based at least in part on associating a predetermined amount of first chunks with a predetermined amount of second chunks; processing the plurality of aligned chunks includes querying the machine learning model based on the first file segment; and the file is classified based, at least in part, on the classification of the first file segment; 22. The system of claim 21.

28. Aligning a predetermined amount of data within a chunk associated with the stream of the file includes: determining an nth file segment based at least in part on associating a predetermined amount of the ith chunk with a predetermined amount of the jth chunk; i and j are positive integers, and j is greater than i; Including, processing the plurality of aligned chunks includes querying the machine learning model based on the nth file segment; and the file is classified based, at least in part, on the classification of the nth file segment; 22. The system of claim 21.

29. The nth file segment contains a predetermined number of bytes.

29. The system of claim 28.

30. the nth file segment contains 1500 bytes; 29. The system of claim 28.

31. the one or more processors are configured to select the predetermined number of bytes from a set of preset numbers of bytes; the predetermined number of bytes is selected based on the packet size of the file; 29. The system of claim 28.

32. Aligning the predetermined amount of data in chunks includes: Adjust the file overhead in the first chunk, and ensuring that sorting involves processing the same number of bytes of the file for each aligned chunk; 22. The system of claim 21.

33. Using the machine learning model to classify the file based on a predetermined number of bytes is deterministic.

22. The system of claim 21.

34. If the prediction obtained from the machine learning model exceeds a predefined maliciousness threshold, the file is determined to be malicious.

22. The system of claim 21.

35. the file is determined to be malicious after the nth chunk is processed using the machine learning model; n corresponds to a positive integer less than the total number of chunks in the file; 35. The system of claim 34.

36. the predetermined maliciousness threshold is constant for each file segment within the file; 35. The system of claim 34.

37. the predetermined maliciousness threshold is dynamic across classifications of file segments within the file; 35. The system of claim 34.

38. In response to determining that the file is malicious, active action is taken against the malicious file.

35. The system of claim 34.

39. The machine learning model is trained using a deep learning process.

22. The system of claim 21.

40. The deep learning process includes a convolutional neural network.

40. The system of claim 39.

41. the convolutional neural network uses a kernel size of less than 12; 41. The system of claim 40.

42. The convolutional neural network uses a kernel size of 8.

41. The system of claim 40.

43. the number of characters for which state information is stored for evaluation of the aligned chunk is based at least in part on a kernel size of the convolutional neural network; 41. The system of claim 40.

44. The machine learning model is an XGBoost model.

22. The system of claim 21.

45. 1. A method for performing classification at an edge device, comprising: obtaining a stream of a file at the edge device; Aligning a predetermined amount of data within a chunk associated with the stream of the file; processing a plurality of aligned chunks associated with the stream of the file using a machine learning model; classifying, at the edge device, the file based at least in part on the classification of the plurality of aligned chunks; A method comprising:

46. 1. A computer program for performing classification at an edge device, the computer program comprising a plurality of instructions stored on a non-transitory computer-readable storage medium, the computer program comprising: The instructions, when executed, cause one or more processors to: obtaining a stream of a file at the edge device; Aligning a predetermined amount of data within a chunk associated with the stream of the file; processing a plurality of aligned chunks associated with the stream of the file using a machine learning model; classifying, at the edge device, the file based at least in part on the classification of the plurality of aligned chunks; To implement Computer program.

Images