Binary file malware detection with structure awareness machine learning

Abstract

Machine learning integrally receives input data from static analysis and dynamic analysis of the binary file to output a maliciousness / goodness determination of the binary file. The machine learning entirety includes structure-aware dynamic compressors ("compressors"). A compressor receives, as input, tree data structures generated based on application programming interface calls of binary files in various sandbox environments. A compressor performs various compression, tokenization, embedding, and shaping operations on the tree data structure to generate a compression tensor that holds a structure context from the tree data structure. Machine learning wholly uses compression tensors to generate malicious / good decisions for binary files.

Description

Technical Field

[0001] This disclosure generally relates to CPC class G06F and subclasses 21 / 50 and / or 21 / 56. Background Technology

[0002] Network security systems utilize static analysis (SA) and dynamic analysis (DA) to analyze binaries for malware detection. SA involves analyzing the code in a binary by examining data such as code patterns, attributes and artifacts, flags and anomalies, without running any of the code. DA involves executing the binary in a sandboxed environment (e.g., a virtual machine) and analyzing its runtime behavior. Sandboxed environments vary, for example, by simulating different operating systems, and DA can execute binaries in multiple sandboxed environments. DA is typically more computationally intensive than SA due to the operations involved in instantiating and dismantling sandboxed environments and running binaries within them. Attached Figure Description

[0003] Embodiments of this disclosure can be better understood by referring to the accompanying drawings.

[0004] Figure 1 This is a schematic diagram of a structure-aware binary malware detection model that includes a structure-aware dynamic compressor with tree-structured DA data.

[0005] Figure 2 This is a schematic diagram of an example structure-aware dynamic compressor.

[0006] Figure 3 Example API call data encoded in a hierarchical pattern and example tree data structures generated from the example API call data are described.

[0007] Figure 4 This is a flowchart of an example operation for detecting malware binaries using structure-aware data transformation.

[0008] Figure 5 This is a flowchart of an example operation for generating dynamically compressed structure-aware tokens from a tree-like data structure.

[0009] Figure 6 This is a flowchart of an example operation for training a structure-aware dynamic compressor and a machine learning model as a whole.

[0010] Figure 7 An example computer system with a structure-aware dynamic compressor and a structure-aware malware detection model is described. Detailed Implementation

[0011] The following detailed description includes example systems, methods, techniques, and procedural flows to aid in understanding this disclosure, and not to limit the scope of the claims. For the sake of brevity, well-known examples of instructions, protocols, structures, and techniques have not been shown in detail.

[0012] Overview The SA and DA of binary files can generate logs with thousands or millions of lines, posing a logical challenge to extracting and analyzing meaningful features from these logs for malware detection. DA logs are API calls to binary files running in an operating system (OS) sandbox, and logs from DA can exhibit hierarchical patterns. Blindly parsing log files to extract strings in the order of appearance and tokenizing the extracted strings using natural language processing (NLP) loses structural awareness of hierarchical patterns, potentially resulting in low-quality input to machine learning (ML) models predicting malware in binary files. This disclosure proposes a structure-aware binary malware detection model (“Model”) incorporating an ML architecture with three pipelines: a first pipeline with SA and DA behavioral data as input, a second pipeline with API call data generated from DA behavioral data represented as a tree data structure as input, and a third pipeline with API call sequences as input. The API call data is used as input to capture hierarchical patterns present in the DA behavioral data. For each binary file, the cloud firewall performs Data Analysis (DA) using multiple sandbox environments targeting multiple operating systems, and the model takes DA data for each OS paired with SA data as input. After three pipelines, the ML architecture includes a max layer and a dense layer. The max layer takes the outputs of the three pipelines from each operating system as input, and the dense layer accepts inputs corresponding to the outputs of the max layer to determine whether the binary file contains malware.

[0013] The second pipeline, which takes tree data structure API call data as input, includes a structure-aware dynamic compressor (“compressor”). The compressor traverses each tree data structure using a tree search algorithm to generate a structure-aware representation of strings at the tree nodes. The compressor then fuses the non-leaf strings in the structure-aware representation to generate a fused string, applies a dictionary mapping to the fused string, and concatenates the dictionary mapping with byte-pair encodings of the leaf strings to generate an irregular tensor. The compressor uses embeddings tuned during model training to embed each row of the irregular tensor corresponding to a path in the tree data structure. Finally, the compressor performs dynamic compression on the irregular tensor using a ratio depending on the height of the irregular tensor relative to a threshold parameter, and reshapes the compressed irregular tensor for input into a convolutional neural network (CNN) in the second pipeline. Structure-aware dynamic compression reduces the input data per binary file while preserving the structural context associated with the tree structure of the input data, thereby producing efficient, high-quality malware predictions at scale.

[0014] the term As used herein, “token” refers to an identifier or value derived from a string. Tokens can include strings extracted from text data separated by delimiter characters (such as whitespace and punctuation) with specific U.S. Standard Code for Information Interchange (ASCII) characters removed, numeric values, numeric embeddings of strings using Natural Language Processing (NLP), any combination thereof, etc.

[0015] As used in this paper, "static compression" refers to compression of a potentially variable-sized input at a fixed compression ratio. In contrast, "dynamic compression," as used in this paper, refers to compression where the compression ratio varies with the input size and other hyperparameters.

[0016] Unless otherwise specified, the use of the phrase "at least one of..." preceding a list with the conjunction "and" should not be considered an exclusive list and should not be interpreted as a list of categories with one item from each category. A clause stating "at least one of A, B, and C" may be violated by only one or more of the listed items, one or more of the items in the list, and other unlisted items.

[0017] Example illustration Figure 1This is a schematic diagram of a structure-aware binary malware detection model that includes a structure-aware dynamic compressor with tree-structured DA data. The structure-aware binary malware detection model (“Model”) 101 includes three pipelines: a first pipeline including a one-hot encoder 103 and a projection layer 105, which takes SA behavior data 114 and DA behavior data 102 as input; a second pipeline including a structure-aware dynamic compressor (“Compressor”) 107 and an API CNN 109, which takes API call data 104 as input; and a third pipeline including an API sequence CNN 117, which takes API sequence data 106 as input, where each of the data inputs is generated for a binary file. The DA behavior data 102, API call data 104, and API sequence data 106 are generated by a cloud firewall 100 using DA in sandbox environments targeting multiple operating systems, and the data from each sandbox environment is paired with the SA behavior data 114 as separate inputs to Model 101—one different input for each operating system. Dense layer 111 receives the connection outputs of the three pipelines and outputs per operating system output 108. Maximum layer 113 receives the connection of per operating system output 108 and feeds it into dense layer 115, which outputs malware determination 112.

[0018] SA behavioral data 114 includes SA data from the code in binary file 190, such as code length data, section length data, number of code sections, digital signatures, the presence of a security directory for portable executable (PE) files, SA determination, etc. DA behavioral data 102 includes data from dynamic code analysis of binary file 190 in the sandbox, such as API call statistics, dynamic link library (DLL) call statistics, identifiers of DA groups running in the sandbox corresponding to the operating system, etc. One-hot encoder 103 converts the category data in SA behavioral data 114 and DA behavioral data 102 into one-hot vectors, and projection layer 105 includes a dense layer that projects the one-hot encoding output by one-hot encoder 103 into vectors for input to dense layer 111.

[0019] API call data 104 includes data from API calls identified during the DA phase of binary file 190. Cloud firewall 100 encodes API call data 104 according to a hierarchical pattern, such as a JavaScript® Object Symbolic (JSON) file. Compressor 107 compresses API call data 104 while using token embedding 121 and byte pair encoding table 119 to preserve the structural context from the hierarchical pattern. Figure 2Details and example hierarchical patterns for compressing API call data 104 are provided. APICNN 109 includes a CNN with 1D convolutional layers and has an architecture that can vary with the number, size, and type of layers depending on the implementation of model 101. API sequence data 106 includes a sequence of API call identifiers and / or call types extracted from API call data 104. Cloud firewall 100 cleans the hierarchical patterns of API call data 104 and extracts the string sequences of API calls in the order of appearance when generating API sequence data 106; in other examples, cloud firewall 100 may generate API sequence data 106 from DA behavior data 102. API sequence CNN 117 includes a CNN with one or more 1D convolutional layers, and this architecture can also vary depending on the implementation with respect to the number, size, and type of layers.

[0020] Each of the three ML pipelines and subsequent layers, including token embedding 121 and byte pair encoding table 119, can be trained as a whole by passing the backpropagation loss through each layer of model 101. Model 101 is trained on SA and DA data (i.e., inputs 114, 102, 104, and 106) of binary files with known malicious / benign labels. Cloud firewall 100 trains model 101 until training criteria are met, such as until the training / test / validation error is sufficiently low, the intrinsic parameters converge across batch training iterations, a threshold number of batches / epochs is reached, etc. Figure 1 The architecture of model 101 described herein is provided as a preferred embodiment for detecting malware binaries. Different architectures and different input formats can be implemented for model 101 (e.g., by removing one or more ML pipelines or adding additional pipelines). For example, the API sequence CNN 117 can alternatively be a recurrent neural network.

[0021] Figure 2This is a schematic diagram of an example structure-aware dynamic compressor. Compressor 107 receives API call data 104 and transforms it into a tree data structure based on its corresponding hierarchical pattern. For example, for a JSON file, compressor 107 can add a child node based on the detection of the "{" character and then traverse to that child node. Based on the detection of the "}" character, the compressor can traverse to the parent node of the current node. Example tree data structure 208 includes a root (i.e., depth 0) node 200A, a depth 1 node 200B, a depth 2 node 200C, and depth 3 nodes 200D and 200E. Each node has an associated string based on fields in the hierarchical pattern. Compressor 107 performs a graph traversal algorithm (e.g., depth-first search) on example tree data structure 208 to generate a sequence of strings at each node along the path, where the strings of leaf nodes in the path are added to the same path sequence, rather than multiple path sequences corresponding to the same parent node depth(s). For example, although leaf nodes 200D and 200E are in different structural paths of the example tree data structure 208, the example path 210 includes a path with nodes 200A, 200B, 200C, 200D, and 200E because they share the same parent node. In some embodiments, the compressor 107 generates the example path 210 while generating the example tree data structure 208. The compressor 107 generates n paths, where n=4 in the depicted example.

[0022] Compressor 107 then merges the strings of each of the example paths 210, where the strings at non-leaf nodes in each path are merged into strings separate from those merged at leaf nodes. In this context, "string merging" refers to concatenating strings sequentially at nodes according to example path 110 to obtain a single string for each sequence of the merged string. The compressor uses a dictionary mapping in token embedding 121 to map the merged strings of non-leaf nodes to tokens, and uses byte-pair encoding table 119 to perform byte-pair encoding on the strings of each leaf node for each path to generate a byte-pair encoded token for each leaf node. In the depicted example, compressor 107 merges the strings at nodes 200A, 200B, and 200C into a single merged string, and uses the dictionary mapping in token embedding 121 to map the merged string to token 202A in example token 214, and performs byte-pair encoding on the strings at nodes 200D and 200E respectively to generate tokens 202B and 202C. The maximum number of tokens per path is k, where k=3 for the depicted example. Due to the variable number of tokens for each path, example token 214 can be stored in a scalable data structure, such as an irregular tensor. In the depicted example, the strings at leaf nodes 200D and 200E are not merged before byte pair encoding, although these strings may be merged in other embodiments.

[0023] Compressor 107 uses the d-dimensional embeddings (d=3 in the depicted example) stored in token embedding 121 to embed each example token 214 to generate example embedding 216. Compressor 107 then dynamically compresses example embedding 216 based on a threshold t (t is the width of the resulting compressed tensor) to generate one of example tensors 218 and 220. The d-dimensional embeddings comprise embedding each entry of the example token 214 into any embedding in the d-dimensional space and can be configured to compress numeric vectors and / or strings, and to dispose of an extended alphabet defined by byte-pair encoding table 119. For example, for numeric vectors with a maximum dimension of m, the d-dimensional embeddings can comprise an m×d matrix whose entries are tuned during training of model 101. The d-dimensional embeddings can differ for byte-pair encoding and dictionary mapping.

[0024] Compressor 107 then statically or dynamically compresses the example embedding 216 at a ratio depending on the relative size of the adjustable threshold parameter t and the number of paths n. The threshold parameter t determines the size of the compressed output of compressor 107. If t > 2n, compressor 107 performs row-by-row dynamic compression on the example embedding 216 to reduce it to an integer upper limit of t / n rows, thereby generating the example tensor 218. For the depicted example, for any t = 9, 10, 11, or 12, the compressor dynamically compresses to an integer upper limit of t / n = 9 / 4 = 3 entries per row. In this case, t is large enough that there is no compression, and the second row of example tensor 218 is filled with entries of the third d dimension. More generally, k can be a larger parameter such that row-by-row compression into a vector with t / n entries reduces the number of entries per row. Compressor 107 dynamically compresses each row by loading the entries into t / n buckets of uniform size and taking the average or maximum value of the entries within each bucket. This disclosure also anticipates other compression algorithms, such as projection, that reduce higher-dimensional vectors to lower-dimensional vectors.

[0025] If t <= 2n, then compressor 107 statically compresses each row of example embedding 216 into a single entry, for example by averaging or taking the maximum value of each row, to generate an example tensor 220 with n rows, where each n row has one entry of dimension d, and then dynamically compresses the row entries of example tensor 220 to generate an example tensor 221 with t rows, each row having one entry of dimension d. Compressor 107 dynamically compresses the row entries by dividing the row entries into t / n buckets and then compressing each bucket as described above.

[0026] For both the cases t > 2n and t <= 2n, compressor 107 shapes the resulting tensors into matrices with d columns and t rows (e.g., by flattening the tensors) for subsequent input into a machine learning model. In the depicted example, the equation t <= 2n is satisfied, and the example tensor 221 resulting from the operation occurring in this case is shaped to generate example matrix 222. In both cases, the resulting example tensors 220 and 221 have a total of dt entries, such that the shaping produces matrices of the same dimension.

[0027] Figure 2Operations such as depth-first search mapping, string merging, tokenization, embedding, and static / dynamic compression are provided as exemplary embodiments for generating compressed representations of paths in the example tree data structure 208. Other methods for generating compressed tree representations, such as adding preprocessing steps, omitting preprocessing steps, combining or changing specific steps (e.g., merging strings of the entire path instead of separating leaf / non-leaf nodes), utilizing different ratios and algorithms for compression, are contemplated in this disclosure. Compressor 107 is configured to generate a compact representation that preserves the hierarchical structure of API call data as captured by the example tree data structure 208, and alternative embodiments may include any compressed / compact representation that preserves the hierarchical structure as opposed to linear parsing methods that parse API calls in the order of appearance without capturing the hierarchical structure. Figure 2 In the example depicted, a single path is shorter than the others, and the corresponding row in the resulting example tensor 218 is padded with zeros. More generally, when generating tensors that are shaped into t×d matrices (such as example tensor 218), any row smaller than the maximum row size is padded with zero entries.

[0028] Figure 3 This describes example API call data encoded in a hierarchical pattern and an example tree-like data structure generated from the example API call data. Example API call data 300 includes: The example API call data 300 is encoded as a JSON file. Different hierarchical patterns can be implemented, and the generation of example API call data 300 can depend on the DA applied to the binary code.

[0029] Example tree data structure 302 generated from example API call data 300 (e.g., by executing DA) Figure 1The component generated in the cloud firewall 100 includes a root node “maec_objects” and child nodes “name:call-library-function”, “api_call”, and “type: malware_action”. The “api_call” node has child nodes “function_name: name1”, “function_name: name2”, and “parameters”. The “parameters” node has child nodes “ProcessHandle:handle1”, “ProcessHandle:handle2”, “ProcessInformationClass:class1”, and “ProcessInformationClass:class2”. The example tree data structure 302 is generated from the example API call data 300 by removing irrelevant syntax such as quotation marks and following a hierarchical pattern to determine child nodes—“{” indicates generating the child nodes of the current node and traversing to that node in the example tree data structure 302, and “}” indicates traversing to the parent node of the current node.

[0030] Figures 4 to 6 This is a flowchart illustrating an example operation for using structure-aware data transformation with an ML architecture to detect malware binaries, and training the structure-aware data transformation and the ML architecture as a whole. To connect with previous... Figure 1 For clarity and / or ease of understanding, the example operations are described with reference to the Structure-Aware Binary Malware Detection Model (“Model”), the Cloud Firewall, and the Structure-Aware Dynamic Compressor (“Compressor”). The names chosen for the program code are not limitations on the claims. The structure and organization of the program may vary due to platform, programmer / architect preferences, programming language, etc. Furthermore, the names of code units (programs, modules, methods, functions, etc.) may vary for the same reasons and can be arbitrary.

[0031] Figure 4 This is a flowchart illustrating example operations for using structure-aware data transformation to detect malware binaries. In box 400, the cloud firewall statically analyzes the malware binary. Binaries include those identified by the cloud firewall for malware detection, such as those detected in storage at endpoint devices (e.g., by a proxy with which the cloud firewall communicates) or those intercepted in the cloud. SA involves analyzing data such as code length data, section length data, the number of code sections, the digital signature of the PE file, the presence of a secure directory, SA determination, etc., without executing the binary.

[0032] In box 402, the cloud firewall determines whether the SA indicates malware. If the SA indicates malware, the action flow proceeds to box 404. Otherwise, the action flow jumps to box 418.

[0033] In box 404, the cloud firewall dynamically analyzes malware binaries within one or more OS sandbox environments. The cloud firewall instantiates the one or more OS sandbox environments (e.g., as virtual machines) and generates logs tracking behavior within each of the one or more sandbox environments as the binary is executed. These logs include runtime behavior data such as API call data, DLL call data, process data, and / or other malware attribute enumeration and characterization (MAEC) objects. Data for API calls, DLL calls, and processes may include timestamps, identifiers, type identifiers, etc.

[0034] In box 406, the cloud firewall begins traversing the operating systems of the sandbox environment where binaries are executed. In box 408, the cloud firewall generates SA data, DA data, API call data in a tree data structure, and API sequence data from the DA for the OS and SA. SA and DA data may include logs from the SA and DA, and the cloud firewall can parse these logs to extract tokens to include in the SA and DA data. When the SA and DA logs become too large, the cloud firewall can trim them (e.g., remove specific types of MAEC objects). DA data is OS-specific, but SA data is uniform across operating systems, and the cloud firewall can generate SA data asynchronously with the iterations that generate DA data for each OS. The logs from the DA of the OS include API call data encoded in a hierarchical pattern, such as JSON files. The cloud firewall generates a tree data structure based on hierarchical relationships between strings indicated in the hierarchical pattern, where each node in the tree data structure corresponds to a string. The API sequence data includes a sequence of identifiers for API calls in the order they appear in the DA logs.

[0035] In box 410, the structure-aware dynamic compressor (“compressor”) generates dynamically compressed structure-aware tokens from a tree-like data structure. (See reference) Figure 5 The operation at box 410 is described in more detail.

[0036] In box 412, the cloud firewall feeds SA and DA data into the first ML pipeline of a structure-aware binary malware detection model (“Model”), feeds dynamically compressed structure-aware tokens into the second ML pipeline of the Model, and feeds API sequence data into the third ML pipeline to obtain a vector of the OS as output. The first ML pipeline includes a one-hot encoder fed into a dense projection layer. The second ML pipeline includes an API CNN, which includes one or more 1D convolutional layers. The third ML pipeline includes an API sequence CNN, which includes one or more 1D convolutional layers. The Model concatenates the outputs of the three ML pipelines and feeds the concatenated outputs into a dense layer to obtain a vector of the OS as output. In box 414, the cloud firewall continues traversing through the operating system. If the binary is executed in a sandboxed environment with an attached OS, the operation flow returns to box 406. Otherwise, the operation flow proceeds to box 416.

[0037] In box 416, the model processes the vectors of each OS at one or more ML layers to obtain a binary malware determination as output. For example, the model can concatenate the vectors of each OS and input this concatenation into a max layer, which is then fed into a dense layer that generates the binary malware determination. The model and compressor, along with the max and dense layers as a whole, are trained using SA and DA data, API call data, and API sequence data of binary files with known malicious / benign determinations. If the binary malware determination indicates that the binary is malicious, the operation flow proceeds to box 418. Otherwise, Figure 4 The operation flow is completed.

[0038] In box 418, the cloud firewall performs corrective actions based on this determination. The severity of the corrective action can vary depending on the credibility of the malicious determination and the context in which the cloud firewall detected the binary file. For example, the cloud firewall may delete the binary file from storage and notify the user and / or administrator of access to the binary file. The cloud firewall may additionally analyze the entity that transmitted the binary file, analyze the execution of the binary file, etc.

[0039] Figure 5This is a flowchart of an example operation for generating dynamically compressed structure-aware tokens from a tree data structure. As a preferred embodiment, the tree data structure is generated from API call data of a DA record in a binary file within a sandbox environment. In block 502, a structure-aware dynamic compressor (“compressor”) traverses the tree data structure using a tree search algorithm to generate a depth-based mapping. For example, the compressor may utilize a depth-first search or a breadth-first search to traverse the tree data structure. The depth-based mapping comprises a list of string sequences of strings at nodes in the tree data structure. Each string sequence includes a path from the root to a non-leaf node in the tree data structure that has leaf children, and one of the sets of leaf nodes of the non-leaf node.

[0040] In box 504, the compressor merges strings from a depth-based mapping to generate a first string for the leaf nodes and a second string for the non-leaf nodes of each string sequence. For example, for each string sequence, the compressor can merge strings from the non-leaf node sequence and strings from the leaf node sequence by concatenating the strings into a single string using whitespace or other delimiting characters.

[0041] In box 506, the compressor byte-pair encodes the first string to generate byte-pair encoded (BPE) tokens for each string, and applies a dictionary mapping to the second string to generate dictionary tokens. Both tokenization operations reduce the size of the first and second strings. The compressor learns a BPE table and a dictionary mapping that maps character sequences to placeholder characters that do not exist as characters in the first string. This dictionary mapping maps strings to tokens during overall training of the compressor using a structure-aware binary malware detection model with backpropagation loss, such as the model described above.

[0042] In box 508, the compressor stores dictionary tokens and BPE tokens as an irregular tensor with n rows, each row containing at most k tokens. The compressor can alternatively utilize any data structure capable of storing variable-size data to store dictionary tokens and BPE tokens. The parameter n is the number of string sequences extracted from the tree-like data structure, and the parameter k is the maximum number of BPE tokens in one of the string sequences plus 1 (where the plus 1 represents the dictionary tokens in the dictionary tokens corresponding to that string sequence).

[0043] In box 509, the compressor embeds tokens into the irregular tensor to generate an embedding tensor. The compressor embeds each token of the irregular tensor in d-dimensional space. Similar to the BPE table and dictionary mapping mentioned earlier, the compressor learns the d-dimensional embeddings during overall training using one of the aforementioned models.

[0044] In box 510, the compressor determines whether t > 2n for a threshold t. t is a parameter that determines the size of the compressed tensor generated from the embedded tensor. If t > 2n, the operation flow proceeds to box 512. Otherwise, the operation flow proceeds to box 514.

[0045] In box 512, the compressor dynamically compresses the embedded tensor row by row at a ratio r = nk / t. The compressor loads each row into an integer upper limit of t / n buckets and compresses each bucket into a single entry. The resulting tensor has n rows and t / n entries, each entry comprising a d-dimensional vector. The operation flow proceeds to box 516.

[0046] In box 514, the compressor statically compresses the embedded tensor row by row at a ratio r = n / t, and dynamically compresses it column by column. First, the compressor compresses each row of the embedded tensor into a single entry, producing a tensor with n rows, each row having a single d-dimensional entry. Then, the compressor packs the resulting columns into t buckets and compresses each bucket, producing a tensor with t rows, each row having a single d-dimensional entry.

[0047] In box 516, the compressor shapes the compressed tensor into a matrix with t rows and d columns. The compressor can shape the compressed tensor by flattening its entries.

[0048] The foregoing disclosure relates to row-wise and column-wise compression of tensors, and in some cases, to compressing rows into single entries. It should be understood that because the foregoing tensors are three-dimensional, compression occurs along the row or column dimension, not along a third embedding dimension represented as a parameter d. For example, for row-wise compression, each entry in a row is a d-dimensional vector. Compression of multiple entries in a row within the same bucket involves an entry-wise compression (e.g., entry-wise maximization or averaging) of the vector of each entry in the d-dimensional vector.

[0049] Figure 6This is a flowchart illustrating example operations for training a structure-aware dynamic compressor (“compressor”) and a machine learning model as a whole. According to the foregoing embodiments, the compressor can be deployed to preprocess inputs to one of three machine learning pipelines. The architecture of the machine learning model can have one or more final layers that take outputs from previous layers corresponding to input data from multiple operating systems. For these embodiments, during training, the final layers backpropagate the loss once across the training data for each operating system and each binary file, while the previous layers comprising the three ML pipelines have losses backpropagated for each operating system. The architecture of the machine learning model can be conceptualized as a stacked copy of three ML pipelines, one copy per OS, and the loss is backpropagated through the first stacked copy. After backpropagation through the first stacked copy, the second stacked copy is updated with new parameters, and then the loss is backpropagated through the second stacked copy, and so on, until the loss is backpropagated for each stacked copy. The backpropagation of the loss can vary depending on the implementation, for example by maintaining separate copies of the machine learning model before one or more final layers corresponding to each operating system.

[0050] In box 602, the machine learning model initializes its parameters at its inner layers, and the compressor initializes the dictionary mapping, token embeddings, and BPE table. The compressor initializes the dictionary mapping as an empty mapping, which will be filled in as new strings are identified during training. The compressor initializes the token embeddings according to a probability distribution (e.g., a Gaussian distribution). Finally, the compressor initializes the BPE table as an empty table, where entries will be filled in as new strings are received by the compressor during training.

[0051] In box 604, the cloud firewall or other entity managing overall training generates training data from the DA and SA of known malicious / benign binaries. The cloud firewall can perform DA against multiple sandbox environments across multiple operating systems. According to the foregoing embodiments, the training data may include SA and DA data, API call data generated from the DA, and API sequence data generated from the DA data. The label of the training data for each binary across potentially multiple operating systems indicates whether the binary is known to be malicious or benign.

[0052] In box 606, the cloud firewall begins iterating through training batches / epochs. In box 608, the cloud firewall backpropagates the loss of the current batch using internal layers of the machine learning model and token embeddings, and updates the byte-pair encoding table and dictionary mapping. Token embeddings include layers of a neural network, and the loss is backpropagated accordingly. The compressor identifies strings not previously seen at the dictionary mapping in its pipeline and initializes new token mappings for the new strings. Token mappings can be randomly generated. In some embodiments, the dictionary mapping also includes layers of a neural network, which are also updated using the backpropagation loss. The compressor additionally determines the character sequence to be replaced with placeholder characters based on the strings of the current batch and updates the BPE table using the mapping between the character sequence and the placeholder characters. Placeholders can alternatively be numeric values.

[0053] In box 610, the cloud firewall determines whether additional training batches / episodes exist. For example, the cloud firewall may determine if termination criteria are met, such as whether the training / test / validation error is sufficiently small, whether a threshold number of batches / episodes has been reached, and whether the intrinsic parameters of the machine learning model and compressor have converged across training iterations. If the cloud firewall determines that additional training batches / episodes exist, the operation flow returns to box 606. Otherwise, the operation flow proceeds to box 612.

[0054] In box 612, the cloud firewall deploys a trained system for malware detection. The cloud firewall then identifies the binary file, applies DA / SA to generate input data for the trained system (potentially across multiple sandbox environments on multiple operating systems), and inputs this data into the trained system to obtain a malicious / benign determination as output.

[0055] change This disclosure differs in that it relates to tree data structures generated from API call data encoded in a hierarchical pattern recorded during the DA phase of a binary file. Alternatively, any of ML and structure-aware dynamic compression techniques can be applied to tree data structures generated from any security-related data encoded in a tree for malware detection. Furthermore, the structure-aware dynamic compression algorithm is extended to any tree data structure other than those generated for network security. The algorithm provided for structure-aware dynamic compression is provided as a preferred embodiment in the context of malware binary file detection. Steps of the algorithm can be changed / omitted, such as the shape of the dynamic / static compression tensor, the type of tokenization, the type of mapping generated from tree traversal, the compression method, etc., and can vary in the implementation. As a simple example, a depth-first search mapping can generate a unique sequence of nodes for each path in the tree data structure, rather than a sequence of nodes that includes multiple paths.

[0056] The flowcharts are provided to aid in understanding the illustrations and are not intended to limit the scope of the claims. The flowcharts depict example operations that can vary within the scope of the claims. Additional operations may be performed; fewer operations may be performed; the operations may be performed in parallel; and the operations may be performed in a different order. For example, the operations depicted in blocks 408, 410, and 412 may be performed in parallel or simultaneously for multiple sandbox operating systems. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by program code. The program code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable machine or device.

[0057] As should be understood, aspects of this disclosure can be embodied as systems, methods, or program code / instructions stored in one or more machine-readable media. Therefore, aspects can take the form of hardware, software (including firmware, resident software, microcode, etc.), or a combination of software and hardware aspects, which are generally referred to herein as “circuit,” “module,” or “system.” The functionality presented as individual modules / units in the example illustrations can be organized differently based on any of the following: platform (operating system and / or hardware), application ecosystem, interface, programmer preferences, programming language, administrator preferences, etc.

[0058] Any combination of one or more machine-readable media may be used. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable storage medium can be, for example, but not limited to, a system, apparatus, or device that uses any one or a combination of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor technologies to store program code. More specific examples (a non-exhaustive list) of machine-readable storage media will include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a machine-readable storage medium can be any tangible medium capable of containing or storing a program used by or in connection with an instruction execution system, apparatus, or device. A machine-readable storage medium is not a machine-readable signal medium.

[0059] Machine-readable signal media may include propagated data signals having machine-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a wide variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. Machine-readable signal media may be any machine-readable medium that is not a machine-readable storage medium and may transmit, propagate, or transfer programs used by or in connection with an instruction execution system, apparatus, or device.

[0060] Program code embodied on a machine-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0061] Program code / instructions may also be stored in a machine-readable medium that can instruct a machine to operate in a particular manner, such that the instructions stored in the machine-readable medium produce an artifact of instructions that include instructions that implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0062] Figure 7 An example computer system with a structure-aware dynamic compressor and a structure-aware malware detection model is depicted. The computer system includes a processor 701 (which may include multiple processors, multiple cores, multiple nodes, and / or implement multithreading, etc.). The computer system includes memory 707. Memory 707 may be system memory or any one or more possible implementations of the machine-readable media described above. The computer system also includes a bus 703 and a network interface 705. The system also includes a structure-aware dynamic compressor (“compressor”) 711 and a structure-aware malware detection model (“model”) 713. Compressor 711 and Model 713 are trained as a whole, and compressor 711 is implemented to preprocess input using structure-aware dynamic compression at one of the three ML pipelines of Model 713. Compressor 711 receives API call data represented in a tree data structure as input, where the API call data is generated from the SA and DA of binary files that additionally produce input data for the other two ML pipelines. According to the foregoing embodiments, compressor 711 generates a depth-first search map from a tree data structure and applies various natural language processing transformations and compression to output a compressed tensor of the tree data structure. Model 713 receives the compressed tensor output by compressor 711 and additional SA / DA data as input and outputs a malicious / benign determination of the binary file. Any of the foregoing functionalities can be implemented partially (or entirely) in hardware and / or processor 701. For example, the functionality can be implemented using application-specific integrated circuits, logic implemented in processor 701, peripheral devices, or coprocessors on cards. Further, implementation may include... Figure 7Fewer or additional components not shown in the diagram (e.g., video card, sound card, additional network interface, peripherals, etc.). Processor 701 and network interface 705 are coupled to bus 703. Although shown as coupled to bus 703, memory 707 may be coupled to processor 701.

Claims

1. A method comprising: Generate a tree-shaped data structure, wherein each node of the tree-shaped data structure includes a corresponding string from a first plurality of strings, wherein the first plurality of strings include strings from dynamic analysis of binary files; A tree search algorithm is used to traverse a tree data structure to generate multiple string sequences from a first plurality of strings, each of which corresponds to one or more paths in the tree data structure, each of which includes a path from the root node to a leaf node of the tree data structure; Compress multiple string sequences to generate multiple compressed representations of a tree data structure; and The multiple compressed representations and the second multiple strings are input into a machine learning system to obtain a determination indicating whether the binary file is malicious or benign as output, wherein the second multiple strings include at least strings generated from static analysis of the binary file.

2. The method of claim 1, wherein the first plurality of strings includes data from an application programming interface call to a dynamically analyzed binary file, wherein the hierarchical structure of the tree data structure corresponds to the hierarchical structure of the data from the application programming interface call.

3. The method according to any one of claims 1-2, wherein compressing the plurality of string sequences comprises: Merge each of the multiple string sequences to generate multiple merged strings; Each of the plurality of fused strings is tokenized to generate multiple tokens; Embed each of the plurality of tokens to generate a plurality of embeddings; as well as Perform at least one of static and dynamic compression on the plurality of embeddings to generate a plurality of compressed representations; Specifically, performing at least one of static and dynamic compression on the plurality of embeddings includes performing at least one of static and dynamic compression on the plurality of embeddings based on a comparison between the number of paths corresponding to the plurality of string sequences in the tree data structure and a threshold parameter. Specifically, the threshold parameter determines the dimension of the plurality of compressed representations.

4. The method according to any one of claims 1-3, Specifically, the first plurality of strings includes dynamically analyzed strings from binaries in sandboxes of one or more operating systems, wherein the tree data structure includes one or more tree data structures for each of the one or more operating systems. Specifically, the machine learning overall includes a first pipeline with the plurality of compressed representations as input and one or more pipelines with the plurality of strings as input. Specifically, the tree search algorithm includes either depth-first search or breadth-first search.

5. A non-transitory computer-readable medium having program code thereon, the program code including instructions for performing the following: Dynamically analyze a binary file to generate a tree data structure, wherein each node in the tree data structure includes a string from a first plurality of strings, wherein each string from the first plurality of strings includes first data generated by the dynamic analysis of the binary file; The tree data structure is compressed to generate multiple compressed representations of the tree data structure, each of the multiple compressed representations including a compressed representation of one of the multiple string sequences, each of the multiple string sequences including a string corresponding to one or more paths in the tree data structure; and The multiple compressed representations and the second data are input into a machine learning system to obtain a determination indicating whether the binary file is malicious or not as output, wherein the second data includes at least data from static analysis of the binary file.

6. The non-transitory computer-readable medium of claim 5, wherein the first plurality of strings comprises data from an application programming interface call to the dynamically analyzed binary file, wherein the hierarchical structure of the tree data structure corresponds to the hierarchical structure of the data from the application programming interface call.

7. The non-transitory computer-readable medium according to any one of claims 5-6, wherein the instructions for compressing the tree data structure include instructions for performing the following: For each of the plurality of string sequences, the string sequences are merged to generate one or more merged strings; Each of the one or more fused strings is tokenized to generate one or more tokens; and Embed each of the one or more tokens to generate one or more embeddings; And perform at least one of static and dynamic compression on the plurality of embeddings to generate a plurality of compressed representations, wherein the plurality of embeddings includes one or more embeddings for each of a plurality of string sequences. Specifically, the instructions for performing at least one of static and dynamic compression on the plurality of embeddings include instructions for performing at least one of static and dynamic compression on the plurality of embeddings based on a comparison between the number of paths corresponding to the plurality of string sequences in the tree data structure and a threshold parameter, wherein the threshold parameter determines the dimension of the plurality of compressed representations.

8. The non-transitory computer-readable medium according to any one of claims 5-7, wherein the first plurality of strings comprises strings from dynamic analysis of binary files in a sandbox of one or more operating systems, wherein the tree data structure comprises one or more tree data structures for each of the one or more operating systems.

9. The non-transitory computer-readable medium according to any one of claims 5-8, wherein the machine learning integral includes a first pipeline having the plurality of compressed representations as input, and one or more pipelines having the second data as input.

10. The non-transitory computer-readable medium according to any one of claims 5-9, wherein the instructions for compressing the tree data structure include instructions for traversing the tree data structure using a tree search algorithm to obtain a plurality of string sequences.

11. An apparatus comprising: processor; and A machine-readable medium storing instructions that can be executed by a processor to cause the device to: Dynamically analyze a binary file to generate a tree-like data structure, wherein each node in the tree-like data structure includes a string from a first plurality of strings, wherein each string from the first plurality of strings includes first data generated by the dynamic analysis of the binary file; The tree data structure is compressed to generate multiple compressed representations of the tree data structure, each of the multiple compressed representations comprising a compressed representation of one of multiple string sequences, each of the multiple string sequences comprising a string corresponding to one or more paths in the tree data structure; and The at least multiple compressed representations are input into a machine learning system to obtain a determination as the output indicating whether the binary file is malicious or benign.

12. The apparatus of claim 11, wherein the first plurality of strings comprises data from an application programming interface call to a dynamically analyzed binary file, wherein the hierarchical structure of the tree data structure corresponds to the hierarchical structure of the data from the application programming interface call.

13. The apparatus according to any one of claims 11-12, wherein the instructions for compressing the tree data structure include instructions executable by the processor to cause the apparatus to: merge the string sequences for each of the plurality of string sequences to generate one or more merged strings; Each of the one or more fused strings is tokenized to generate one or more tokens; and Embed each of one or more tokens to generate one or more embeddings; as well as At least one of static and dynamic compression is performed on the plurality of embeddings to generate a plurality of compressed representations, wherein the plurality of embeddings includes one or more embeddings for each of a plurality of string sequences. Specifically, the instructions for performing at least one of static and dynamic compression on the plurality of embeddings include instructions executable by a processor to cause the device to perform at least one of static and dynamic compression on the plurality of embeddings based on a comparison between the number of paths corresponding to the plurality of string sequences in the tree data structure and a threshold parameter, wherein the threshold parameter determines the dimension of the plurality of compressed representations.

14. The apparatus of any one of claims 11-13, wherein the first plurality of strings comprises strings from dynamically analyzed binary files in a sandbox of one or more operating systems, wherein the tree data structure comprises one or more tree data structures for each of the one or more operating systems.

15. The apparatus of any one of claims 11-14, wherein the machine learning overall comprises a first pipeline having the plurality of compressed representations as input, and one or more pipelines having second data as input, wherein the second data comprises at least data from static analysis of the binary file.

Images