Automatic code signing generation for windows.net binaries

Abstract

Various embodiments provide a system, method, and apparatus for generating signatures for Windows .NET binary files. The method includes (i) generating a file signature based on code using hashing techniques, and (ii) classifying samples based on the code using the file signature.

Description

BACKGROUND

[0001] Unscrupulous individuals attempt to compromise computer systems in various ways. As one example, such individuals can embed or otherwise include malware in an email attachment and transmit or cause the malware to be transmitted to unsuspecting users. When executed, the malware can compromise the victim's computer. Certain types of malware can instruct the compromised computer to communicate with a remote host. For example, the malware can turn the compromised computer into a "zombie" in a "botnet" that receives instructions from and / or reports data to a command and control (C&C) server under the control of the unscrupulous individual. One approach to mitigating damage caused by malware is for a security company (or other appropriate entity) to attempt to identify the malware and prevent it from reaching / being executed on end user computers. Another approach is to attempt to prevent the compromised computer from communicating with the C&C server. Unfortunately, malware authors are using increasingly sophisticated techniques to obfuscate the workings of their software. As such, there is a continuing need for improved techniques to detect malware and prevent it from doing harm. BRIEF DESCRIPTION OF DRAWINGS

[0002] Various embodiments of the present invention are disclosed in the detailed description and drawings below.

[0003] Figure 1 is a block diagram of an environment in which a malicious file is detected or suspected in accordance with various embodiments.

[0004] Figure 2 is a block diagram of a system that detects malicious files in accordance with various embodiments.

[0005] Figure 3A is a code illustration of a method contained in a benign.NET file example.

[0006] Figure 3B is a schematic of a wildcarded method based on the method code of the method contained in the benign.NET file example.

[0007] Figure 3C is a code illustration of a method contained in a benign.NET file example.

[0008] Figure 3D is a schematic of a wildcarded method based on the method code of the method contained in the benign.NET file example.

[0009] Figure 4A is a code illustration of a method contained in a malicious.NET file example.

[0010] Figure 4BThis is a diagram illustrating the wildcarding method, which is based on the method code of a method included in a malicious .NET file example.

[0011] Figure 4C This is the code description of a method included in a malicious .NET file example.

[0012] Figure 4D This is a diagram illustrating the wildcarding method, which is based on the method code of a method included in a malicious .NET file example.

[0013] Figure 5 This is a flowchart of a method for identifying file families based on file signatures according to various embodiments.

[0014] Figure 6 This is a flowchart of a method for classifying samples based at least in part on the file signatures of the samples, according to various embodiments.

[0015] Figure 7 These are examples of transforming function code into wildcard functions according to various implementation methods.

[0016] Figure 8 This is a flowchart of a method for performing sample classification according to various embodiments.

[0017] Figure 9 This is a flowchart of a method for performing sample classification according to various embodiments.

[0018] Figure 10 This is a flowchart of a method for determining an intermediate hash of a function within the code for a sample, according to various embodiments.

[0019] Figure 11 This is a flowchart of a method for detecting malicious samples based at least in part on an intermediate hash of a function within the code used for the sample, according to various embodiments.

[0020] Figure 12 This is a flowchart of a method for obtaining a file signature for a sample, according to various embodiments.

[0021] Figure 13 This is a flowchart of a method for clustering files based on their respective file signatures to obtain a set of file families, according to various embodiments.

[0022] Figure 14 This is a flowchart of a method for obtaining analysis results for a set of file families according to various embodiments.

[0023] Figure 15 This is a flowchart of a method for detecting malicious files in received traffic according to various embodiments.

[0024] Figure 16 This is a flowchart of a method for classifying and processing files received in network traffic, according to various embodiments. Detailed Implementation

[0025] This invention can be implemented in a variety of ways, including as a method; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided thereto in memory coupled to the processor. In this specification, these implementations or any other form of the invention that may take the form of a technique may be referred to as such. Generally, within the scope of this invention, the order of steps of the disclosed processes may be varied. Unless otherwise stated, components such as processors or memory described as configured to perform a task may be implemented as general-purpose components temporarily configured to perform the task at a given time or manufactured as specific components to perform that task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores, such as computer program instructions, configured to process data.

[0026] The following provides a detailed description of one or more embodiments of the present invention, along with accompanying drawings illustrating the principles of the invention. The invention is described in conjunction with these embodiments, but is not limited to any particular embodiment. The scope of the invention is limited only by the claims, which include numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention can be practiced according to the claims without requiring some or all of these specific details. For clarity, technical materials known in the art related to the invention have not been described in detail so that the invention is not unnecessarily obscured.

[0027] As used in this article, a security entity is a network node (e.g., a device) that enforces one or more security policies on information such as network traffic and files. For example, a security entity can be a firewall. As another example, a security entity can be implemented as a router, switch, DNS resolver, computer, tablet, laptop, smartphone, etc. Various other devices can be implemented as security entities. As yet another example, security can be implemented as an application running on the device, such as an anti-malware application.

[0028] As used herein, malware refers to an application that performs actions, whether clandestine or illegal, that a user would not approve / would not authorize if fully aware of them. Examples of malware include Trojans, viruses, rootkits, spyware, hacking tools, keyloggers, etc. One example of malware is a desktop application that collects and reports the end-user's location to a remote server (but does not provide the user with location-based services such as map services). Another example of malware is a malicious Android Application Package .apk (APK) file that appears to the end-user as a free game but secretly sends text messages (e.g., costing $10 per message), increasing the end-user's phone bill. An example of malware is the Apple iOS flashlight app, which secretly collects the user's contacts and sends them to spammers. Other forms of malware (e.g., ransomware) can also be detected / blocked using the techniques described herein. Furthermore, although malware signatures are described herein as being generated for malicious applications, the techniques described herein can also be used in various embodiments to generate profiles for other types of applications (e.g., adware profiles, product profiles, etc.).

[0029] Binaries generated by the .NET Framework contain a wealth of information for malware learning and detection. For example, imported DLLs, functions, and TypeRef tables in .NET binaries can be hashed and used as file signatures, known as TypeRefHash or TRH. However, TRH relies on DLL symbols and function names, which are easily obfuscated to evade detection. In contrast, code obfuscation in .NET binaries written in Microsoft Intermediate Language (MSIL) is less common and more difficult. Furthermore, because code is a direct representation of the functionality of an executable file, malware detection that relies on code is generally more accurate. Therefore, various implementations automatically transform the code of each method in a .NET binary into a hash, based on which a file signature (also referred to herein as DNCodeHash) is calculated. File signatures can be used for malware detection with low false positive rates and for malware clustering based on code similarity.

[0030] According to various embodiments, the system (i) parses a .NET binary and identifies methods (e.g., functions called by the file's code), (ii) transforms the MSIL code of each method into a uniform format and hashes that uniform format (e.g., to obtain a function hash or intermediate signature), and (iii) combines a set of function hashes (e.g., all function hashes of a file) into a single file signature (e.g., DNCodeHash). DNCodeHash can be used for malware learning, detection, and clustering. Furthermore, function hashes can be used as a threat indicator in more fine-grained malware detection. Parsing can rely on various open-source tools. .NET binaries typically conform to well-defined formats and MSIL, and the system disassembles the .NET binary and parses the format and code. Various parsing techniques can be implemented. For example, the system can use predefined libraries such as the Python libraries dnfile and dncil. During the MSIL code transformation, for each MSIL instruction in a method, the system wildcards its operands. Therefore, each method becomes data-independent. In this way, the wildcarded representation provides the signature of the method. In the application, each wildcard-based method representation is hashed (e.g., to obtain an intermediate signature or function hash) for efficient storage and comparison. All hashes are then sorted, concatenated, and hashed again into a composite hash, DNCodeHash. DNCodeHash can be used, similar to TypeRefHash, as an indicator of inconsistency of control (IOC).

[0031] According to various embodiments, wildcard representations for .NET methods achieve high precision in the abstract function code schema, which can be used as function-level code signatures. Furthermore, file signatures (e.g., DNCodeHash) have high precision in the abstract global code schema, which can be used as an Inversion of Control (IOC) for malware detection. Signatures generated from a single sample can cover a large number of new samples within the same family. Experimental data shows that DNCodeHash can reduce false positives by 34.3% and false negatives by 16.1% when used as an IOC. 3. DNCodeHash is effective in code similarity-based clustering. It can improve basic fact-finding and malware family analysis, and help improve the efficiency of analyzing false positives and false negatives. File clustering based on DNCodeHash can be used to discover new malware families. In addition to the above, DNCodeHash signature generation is fully automated and efficient. For example, a system according to various embodiments can determine the file signature (e.g., DNCodeHash) of a .NET binary file in 0.518 seconds.

[0032] Various embodiments provide a system, method, and apparatus for generating signatures for Windows .NET binary files. The method includes (i) generating a file signature based on code using hashing techniques, and (ii) classifying samples based on the code using the file signature.

[0033] Figure 1 This is a block diagram of an environment in which malicious files are detected or suspected, according to various embodiments. In the example shown, client devices 104-108 (respectively) are laptops, desktop computers, and tablets residing in corporate network 110 (belonging to "Acme Corporation"). Data device 102 is configured to implement policies (e.g., security policies) regarding communication between client devices such as client devices 104 and 106 and nodes outside corporate network 110 (e.g., reachable via external network 118). Examples of such policies include policies managing traffic shaping, quality of service, and traffic routing. Other examples of policies include security policies such as requiring the scanning of incoming (and / or outgoing) email attachments, website content, input to application portals (e.g., web interfaces), and threats in file and / or other file transfers exchanged via instant messaging programs. In some embodiments, data device 102 is also configured to enforce policies regarding traffic remaining within (or entering) corporate network 110.

[0034] The techniques described in this article can be used in conjunction with various platforms (e.g., desktops, mobile devices, gaming platforms, embedded systems, etc.) and / or various types of applications (e.g., Android .apk files, iOS applications, Windows PE files, Adobe Acrobat PDF files, Microsoft Windows PE installers, .NET files, etc.). Figure 1 In the example environment shown, client devices 104-108 are a laptop, a desktop computer, and a tablet computer located within corporate network 110, respectively. Client device 120 is a laptop computer located outside corporate network 110.

[0035] Data device 102 can be configured to work in conjunction with a remote security platform 140. Security platform 140 can provide various services, including performing static and dynamic analysis on malware samples, providing a list of signatures of known exploits (e.g., malicious input strings, malicious files, etc.) as part of a subscription to data devices such as data device 102, detecting exploits such as malicious input strings or malicious files (e.g., on-demand detection, or periodic updates to file mappings to indicate whether a file is malicious or benign), providing the probability that an input string or file (e.g., a .NET file) is malicious or benign, providing / updating a whitelist of input strings or files considered benign, providing / updating input strings and files defined as malicious, identifying malicious domains, detecting malicious input strings, detecting malicious files, predicting whether an input string or file (e.g., a .NET file) is malicious, and providing an indication that an input string or file is malicious (or benign). In various embodiments, the analysis results (and additional information related to applications, domains, etc.) are stored in database 160. In various embodiments, security platform 140 includes one or more dedicated commercial hardware servers (e.g., with multi-core processors, 32GB+ RAM, gigabit network interface adapters, and hard drives) running a typical server-class operating system (e.g., Linux). Security platform 140 can be implemented across a scalable infrastructure including multiple such servers, solid-state drives, and / or other suitable high-performance hardware. Security platform 140 may include multiple distributed components, including components provided by one or more third parties. For example, some or all of security platform 140 may be implemented using Amazon Elastic Compute Cloud (EC2) and / or Amazon Simple Storage Service (S3). Furthermore, as with data device 102, whenever security platform 140 is referred to as performing a task, such as storing or processing data, it should be understood that one or more sub-components of security platform 140 (whether individually or in collaboration with third-party components) may cooperate to perform that task. As an example, security platform 140 may optionally collaborate with one or more virtual machine (VM) servers to perform static / dynamic analysis. An example of a virtual machine server is a physical machine comprising commercially available server-grade hardware (e.g., a multi-core processor, 32+ gigabytes of RAM, and one or more gigabit network interface adapters) running commercially available virtualization software such as VMware ESXi, Citrix XenServer, or Microsoft Hyper-V. In some embodiments, the virtual machine server is omitted. Furthermore, the virtual machine server may be under the control of the same entity managing the security platform 140, or it may be provided by a third party.As an example, the virtual machine server may rely on EC2, and the rest of the security platform 140 is provided by dedicated hardware owned and controlled by the security platform 140 operator.

[0036] In some embodiments, system 100 (e.g., malicious file detector 170, security platform 140, etc.) is configured to detect exploits (e.g., malicious .NET files), such as by generating a file signature and querying the signature index of known files (e.g., previously classified files), or by assigning the file to another system or service (e.g., a machine learning model or operator) responsible for analyzing and classifying the file. System 100 identifies a set of functions in the file and generates a file signature based at least in part on that set of functions, such as based on multiple intermediate signatures computed respectively for the various functions in that set.

[0037] According to various embodiments, security platform 140 includes a DNS tunnel detector 138 and / or a malicious file detector 170. The malicious file detector 170 is used to determine whether a .NET file is malicious. In response to receiving a sample (e.g., a .NET file), the malicious file detector 170 analyzes the .NET file and determines whether the sample is malicious. For example, the malicious file detector 170 determines the file signature and queries a signature index / database of known files (e.g., files previously analyzed and categorized), such as a whitelist or blacklist of files. In some embodiments, the malicious file detector 170 receives a .NET file, performs function extraction, determines the intermediate signature of the extracted function, determines the file signature based on the intermediate signature, and determines (e.g., predicts) whether the .NET file is malicious, at least in part, based on querying an index of known files (e.g., a mapping of known file signatures to corresponding indications of whether a file is malicious).

[0038] In some embodiments, the malicious file detector 170 includes one or more of a .NET file parser 172, a signature generation module 174, a prediction engine 176, and / or a cache 178.

[0039] The .NET file parser 172 is used to obtain information related to the sample, such as the .NET file. In some embodiments, the .NET file parser 172 disassembles the file to obtain the MSIL and identifies the various functions called by the file.

[0040] The signature generation module 174 generates a file signature for the file. In some embodiments, the signature generation module 174 generates the file signature based at least in part on a set of identified functions called by the file. For example, the signature generation module 174 determines (i) the function code for the identified functions, (ii) function wildcards based on the function code (e.g., a wildcardization method), (iii) a function hash / intermediate signature for the function wildcards, and (iv) a file signature based on the function hash / intermediate signature.

[0041] In some embodiments, the prediction engine 176 determines whether the .NET is malicious based at least in part on one or more of the following (e.g., providing a prediction / probability that the file is malicious): (i) a mapping of the .NET file to an indication of whether the corresponding .NET file is malicious, (ii) a mapping of the identifier of the .NET file (e.g., a hash or other signature associated with the .NET file) to an indication of whether the corresponding .NET file is malicious, and / or (iii) a classifier (e.g., a model trained using a machine learning process).

[0042] According to various embodiments, in response to prediction engine 176 determining that a received .NET file is malicious, the system sends an indication that the .NET file is malicious to a security entity. For example, malicious file detector 170 may send the indication that the .NET file is malicious to a security entity (e.g., a firewall) or a network node (e.g., a client). The indication that the .NET file is malicious may correspond to an update to a blacklist of .NET files (e.g., for malicious .NET files) if the received .NET file is considered malicious, or an update to a whitelist of .NET files if the received .NET file is considered benign. In some embodiments, malicious file detector 170 combines the indication that the .NET file is malicious or benign with a hash or signature corresponding to the .NET file. The security entity or endpoint may compute the hash or signature of the .NET file and perform a lookup (e.g., querying whitelists and / or blacklists) against the mapping of the hash / signature to the indication that the .NET file is malicious / benign. In some embodiments, the hash or signature uniquely identifies the .NET file.

[0043] In response to receiving a .NET file to be analyzed, the malicious file detector 170 may determine whether the .NET file corresponds to a previously analyzed .NET file (e.g., whether the .NET file matches a .NET file associated with historical information previously used to determine malice). For example, the malicious file detector 170 determines whether historical information (e.g., blacklists, whitelists, etc.) includes an identifier or representation information corresponding to the .NET file. In some embodiments, the representation information corresponding to the .NET file is a hash or signature of the .NET file. In some embodiments, the malicious file detector 170 (e.g., prediction engine 176) determines whether information related to a particular .NET file is included in a dataset of historical .NET files and / or in historical information associated with historical datasets indicating whether a particular .NET file is malicious (e.g., third-party services such as VirusTotal™). In response to determining that information relating to a particular .NET file is not included in or available in a dataset of historical .NET files and / or historical information, the malicious file detector 170 may assume that the .NET file has not yet been analyzed, and the malicious file detector 170 may invoke analysis of the .NET file in conjunction with determining (e.g., predicting) whether the .NET file is malicious (e.g., the malicious file detector 170 may combine the determination of whether the .NET file is malicious with a .NET file query classifier). An example of historical information associated with historical files indicating whether a particular .NET file is malicious corresponds to a VirusTotal® (VT) score. If a particular .NET file has a VT score greater than 0, a third-party service will consider that particular .NET file to be malicious. In some embodiments, historical information associated with historical .NET files indicating whether a particular .NET file is malicious corresponds to social ratings, such as community-based ratings or scores (e.g., reputation scores), indicating that the .NET file is malicious or likely to be malicious. Historical information (e.g., from third-party services, community-based ratings, etc.) indicates whether other vendors or cybersecurity organizations consider a particular .NET file to be malicious.

[0044] In some embodiments, a malicious file detector 170 (e.g., a prediction engine 176) determines that the received file is newly analyzed (e.g., the .NET file is not in historical information / datasets, not on a whitelist or blacklist, etc.). In response to security platform 140 receiving a .NET file from a security entity (e.g., a firewall) or endpoint within the network, malicious file detector 170 (e.g., a .NET file parser 172) may detect that a new .NET file has been analyzed. For example, malicious file detector 170 determines whether the .NET file was newly analyzed simultaneously with a .NET file being received by security platform 140 or malicious file detector 70. As another example, malicious file detector 170 (e.g., prediction engine 176) determines that a new .NET file has been analyzed based on a predefined schedule (e.g., daily, weekly, monthly, etc.), such as in conjunction with batch processing. In response to determining that the received .NET file has not yet been analyzed regarding whether such a .NET file is malicious (e.g., the system does not include historical information related to such a .NET file), the malicious file detector 170 determines whether another service was used to analyze the .NET file, such as a machine learning model (e.g., querying a classifier to analyze a .NET file or one or more feature vectors associated with a .NET file) or a human operator.

[0045] Cache 178 stores information related to .NET files. In some embodiments, cache 178 stores a mapping from whether a .NET file is malicious (or potentially malicious) to a specific .NET file, or a mapping from a .NET file being malicious or potentially malicious to a hash or signature corresponding to the .NET file. Cache 178 may store additional information related to a set of .NET files, such as attributes of the .NET files, hashes or signatures corresponding to the .NET files in that set, other unique identifiers corresponding to one of the .NET files in that set, etc.

[0046] return Figure 1 Suppose a malicious individual (using client device 120) creates malware 130 (e.g., a malicious .NET file). The malicious individual wants client devices (such as client device 104) to execute a copy of malware or other exploits (e.g., a malicious .NET file) 130, compromise the client devices, and turn them into zombies in a botnet. The compromised client devices can then be instructed to perform tasks (e.g., cryptocurrency mining, or participating in denial-of-service attacks) and / or report information to external entities (e.g., those associated with such tasks, such as leaking sensitive corporate data), such as command and control (C&C) server 150, and receive instructions from C&C server 150 (if applicable).

[0047] Figure 1 The environment shown includes three Domain Name System (DNS) servers (122-126). As shown, DNS server 122 is under the control of ACME (for use by computing assets located within corporate network 110), while DNS server 124 is publicly accessible (and can also be used by computing assets located in network 110 and other devices, such as devices located in other networks (e.g., networks 114 and 116)). DNS server 126 is publicly accessible but under the control of a malicious operator of C&C server 150. Corporate DNS server 122 is configured to resolve corporate domain names to IP addresses and is also configured to communicate with one or more external DNS servers (e.g., DNS servers 124 and 126) to resolve domain names where applicable.

[0048] As described above, in order to connect to a legitimate domain (e.g., www.example.com, described as website 128), a client device, such as client device 104, will need to resolve the domain to its corresponding Internet Protocol (IP) address. One way this resolution can occur is that client device 104 forwards the request to DNS servers 122 and / or 124 to resolve the domain. In response to receiving a valid IP address for the requested domain name, client device 104 can use that IP address to connect to website 128. Similarly, in order to connect to a malicious C&C server 150, client device 104 will need to resolve the domain “kj32hkjqfeuo32ylhkjshdflu23.badsite.com” to its corresponding Internet Protocol (IP) address. In this example, malicious DNS server 126 is authoritative for *.badsite.com, and client device 104's request will be forwarded (e.g.) to DNS server 126 for resolution, ultimately allowing C&C server 150 to receive data from client device 104.

[0049] Data device 102 is configured to implement policies regarding communication between client devices such as client devices 104 and 106 and nodes outside the corporate network 110 (e.g., reachable via external network 118). Examples of such policies include those managing traffic shaping, quality of service, and traffic routing. Other examples of policies include security policies such as requiring the scanning of incoming (and / or outgoing) email attachments, website content, information entered into web interfaces such as login screens, files and / or other file transfers exchanged via instant messaging programs, and / or isolating or deleting files or other exploits identified as malicious (or potentially malicious). In some embodiments, data device 102 is also configured to implement policies regarding traffic remaining within the corporate network 110.

[0050] In various embodiments, data device 102 includes a DNS module 134 configured to help determine whether client devices (e.g., client devices 104-108) are attempting to participate in a malicious DNS tunnel, and / or block (e.g., connections by client devices 104-108) to a malicious DNS server. The DNS module 134 may be integrated into data device 102 (e.g., ...). Figure 1 (as shown), and in various embodiments can also operate as a standalone device. Furthermore, with Figure 1 Like the other components shown, DNS module 134 can be provided by the same entity that provides data device 102 (or security platform 140), or by a third party (e.g., a third party different from the provider of data device 102 or security platform 140). Furthermore, in addition to preventing connections to malicious DNS servers, DNS module 134 can take other actions, such as personalizing the logging of tunneling attempts made by clients (indicating that a client has been compromised and should be isolated, or otherwise investigated by an administrator).

[0051] In various embodiments, when a client device (e.g., client device 104) attempts to resolve a domain, the DNS module 134 uses that domain as a query to the security platform 140. This query can be performed concurrently with domain resolution (e.g., concurrently with requests sent to DNS servers 122, 124, and / or 126 and the security platform 140). As an example, the DNS module 134 can send a query (e.g., in JSON format) to the front end 142 of the security platform 140 via a REST API. Using the processing described in more detail below, the security platform 140 will (e.g., using a DNS tunnel detector 138) determine whether the queried domain indicates a malicious DNS tunneling attempt and provide the result back to the DNS module 134 (e.g., "malicious DNS tunnel" or "non-tunnel").

[0052] In various embodiments, when a client device (e.g., client device 104) attempts to open a .NET file, such as an email attachment, instant message, or otherwise received via network exchange, or when the client device receives such a .NET file, the DNS module 134 uses the .NET file (or a computed hash or signature, or other unique identifier, etc.) as a query to the security platform 140. This query can be executed simultaneously with the receipt of the .NET file or in response to a request from a user to scan for .NET files. As an example, data device 102 can send a query (e.g., in JSON format) to the front end 142 of the security platform 140 via a REST API. Using the processing described in more detail below, the security platform 140 will (e.g., using a malicious file detector, possibly similar to malicious file detector 170, such as by using a machine learning model to detect / predict whether a file is malicious) determine whether the queried file is malicious (or likely to be malicious) and provide the result back to data device 102 (e.g., "malicious file" or "benign file").

[0053] In various embodiments, the DNS tunnel detector 138 (whether implemented on security platform 140, data device 102, or other suitable location / combination) uses a two-pronged approach to identify malicious DNS tunnels. The first approach uses an anomaly detector 146 (e.g., implemented in Python) to construct a live profile (156) of a set of DNS traffic for the root domain. The second approach uses signature generation and matching (also referred to herein as similarity detection, and implemented, for example, in Go). These two approaches are complementary. The anomaly detector serves as a general detector that can identify previously unknown tunnel traffic. However, the anomaly detector may need to observe multiple DNS queries before detection occurs. To block the first DNS tunnel packet, a similarity detector 144 complements the anomaly detector 146 and extracts a signature from the detected tunnel communication that can be used to identify cases where an attacker has registered a new malicious tunnel root domain but has done so using tools / malware similar to the detected root domain.

[0054] When data device 102 receives a DNS query (e.g., from DNS module 134), data device 102 provides the DNS query to security platform 140, which performs anomaly detection and similarity detection, respectively. In various embodiments, if either detector flags a domain (e.g., as provided in a query received by security platform 140), that domain is classified as a malicious DNS tunnel root domain.

[0055] The DNS tunnel detector 138 maintains a set of fully qualified domain names (FQDNs) for each device (from which it receives data), and these domain names are grouped according to their root domain (in...Figure 1 (This is illustrated in Domain Profile 156). (Although grouping by root domain is generally described in this specification, it should be understood that the techniques described herein can also be extended to domains at any level.) In various embodiments, information about received queries for a given domain is maintained in the profile for a fixed amount of time (e.g., a sliding time window of ten minutes).

[0056] In some embodiments, the malicious file detector 170 provides an indication to a security entity, such as data device 102, of whether a .NET file is malicious. For example, in response to determining that a .NET file is malicious, the malicious file detector 170 sends the indication that the .NET file is malicious to data device 102, and the data device may then implement one or more security policies based at least in part on the indication that the .NET file is malicious. One or more security policies may include isolating / separating .NET files, deleting .NET files, ensuring that .NET files are not executed or parsed, warning or prompting the user about the maliciousness of a .NET file before the user opens / executes it, etc. As another example, in response to determining that a .NET file is malicious, the malicious file detector 170 provides the security entity with an update to the mapping of .NET files (or hashes, signatures, or other unique identifiers corresponding to .NET files) to the corresponding indication of whether a .NET file is malicious, or an update to a blacklist of malicious .NET files (e.g., identifying malicious .NET files) or a whitelist of benign .NET files (e.g., identifying .NET files not considered malicious).

[0057] Figure 2 This is a block diagram of a system for detecting malicious files according to various embodiments. According to various embodiments, system 200 combines... Figure 1 System 100 is used to implement this, such as for malicious file detector 170. In various embodiments, system 200 is combined with Figure 5 Process 500 Figure 6 Process 600 Figure 8 The process 800 Figure 9 The process 900 Figure 10 Process 1000 Figure 11 Process 1100 Figure 12 Process 1200 Figure 13 Process 1300 Figure 14 Process 1400 Figure 15 Process 1500 and / or Figure 16 The process 1600 is implemented. System 200 can be implemented in one or more servers, security entities such as firewalls, and / or endpoints.

[0058] System 200 can be implemented using one or more devices, such as a server. System 200 can be implemented at different locations on a network. In some embodiments, system 200 is implemented... Figure 1 System 100 includes a malicious file detector 170. As an example, system 200 is deployed as a service such as a web service (e.g., system 200 determines whether a .NET file is malicious and provides this determination as a service). This service may be provided by one or more servers (e.g., system 200 or the malicious file detector is deployed on a remote server that monitors or receives files (e.g., .NET files) transmitted within or to / from the network, determines whether the .NET file is malicious, and sends / pushes notifications or updates related to the .NET file, such as indications of whether the .NET file is malicious, for example, via input to a web interface such as a login screen, authentication interface, query interface, or attachments to email, instant messages, etc.). As another example, the malicious file detector is deployed on a firewall.

[0059] According to various embodiments, in response to receiving a .NET file to be analyzed to determine whether the .NET file is malicious, system 200 uses a classifier to determine whether the .NET file is malicious (or to determine the probability that the .NET file is malicious). For example, system 200 uses a classifier to provide a prediction of whether the .NET file is malicious. In some embodiments, system 200 determines the file signature of the .NET file and uses the classifier to analyze the file signature in conjunction with the determination of whether the .NET file is malicious. For example, system 200 uses the file signature to query an index (e.g., a mapping of signatures to corresponding indications of whether the file is malicious, benign, etc.) to determine whether the index stores matching signatures of known samples (e.g., samples previously analyzed / classified).

[0060] In the example shown, system 200 incorporates one or more modules for classifying files, such as predicting whether a .NET file (e.g., a newly received .NET file) is malicious, determining the likelihood that a .NET file is malicious, and / or providing notification or indication that a .NET file is malicious or benign / non-malicious. System 200 includes a communication interface 205, one or more processors 210, a storage device 215, and / or a memory 220. The one or more processors 210 include one or more of the following: a communication module 225, a .NET extraction module 227, a function extraction module 229, a function wildcard module 231, an intermediate signature module 233, an intermediate classifier module 235, a file signature determination module, a classification module 239, a clustering module 241, an allocation module 243, a notification module 245, and a secure execution module 247.

[0061] In some embodiments, system 200 includes a communication module 225. System 200 uses communication module 225 to communicate with various nodes or endpoints (e.g., client terminals, firewalls, DNS resolvers, data devices, other security entities, etc.) or user systems such as administrator systems. For example, communication module 225 provides information to be transmitted (e.g., to another node, security entity, etc.) to communication interface 205. As another example, communication interface 205 provides information received by system 200 to communication module 225. Communication module 225 is configured to receive .NET files to be analyzed, such as from network endpoints or nodes, such as security entities (e.g., firewalls), database systems, query systems, etc. Communication module 225 is configured to query third-party services for information related to .NET files (e.g., services that publicly disclose .NET file information, such as third-party ratings or malice assessments of .NET files, community-based ratings, assessments or reputations associated with .NET files, blacklists and / or whitelists of .NET files, etc.). For example, system 200 uses communication module 225 to query third-party services. Communication module 225 is configured to receive one or more settings or configurations from an administrator. Examples of one or more settings or configurations include configurations for determining whether a .NET file is a malicious process, determining the format or process upon which a combined feature vector is based, a set of feature vectors to be provided to a classifier used to determine whether a .NET file is malicious, a definition of a hash function used to determine an intermediate signature, a definition of a hash function used to determine a file signature, configurations related to the sample traffic (e.g., benign or malicious files) used to generate the training classifier, information related to a whitelist of .NET files (e.g., .NET files not considered suspicious and whose traffic or attachments are allowed), information related to a blacklist of .NET files (e.g., .NET files considered suspicious and whose traffic will be quarantined, deleted, or otherwise restricted from execution), and so on.

[0062] In some embodiments, system 200 includes a .NET extraction module 227. System 200 uses the .NET extraction module 227 in conjunction with determining whether to extract information related to the header or metadata of a .NET file (e.g., from) and extracting information from the .NET file (e.g., for analyzing whether the .NET file is malicious). In some embodiments, the .NET extraction module 227 receives a .NET file to be analyzed, such as a .NET file included as an attachment to an email or instant message, or a .NET file otherwise transmitted across or to / from a network. The .NET extraction module disassembles the .NET file to obtain the MSIL code corresponding to the .NET file to be analyzed.

[0063] In some embodiments, system 200 includes a function extraction module 229. System 200 uses function extraction module 229 to identify functions in MSIL code obtained by .NET extraction module 227. In response to identifying functions in MSIL code, the system extracts the corresponding function code (e.g., also referred to herein as method code).

[0064] In some embodiments, system 200 includes a function wildcard module 231. System 200 uses function wildcard module 231 to obtain function wildcards from the function code obtained by function extraction module 229. Function wildcard module 231 obtains wildcards by transforming the function into a data-independent representation. For example, function wildcard module 231 determines the operators and corresponding operands in the function and replaces the operands with predefined special characters (e.g., ?). Function wildcard module 231 normalizes the function to ignore operand information.

[0065] In some embodiments, system 200 iteratively obtains function wildcards, obtains intermediate signatures for the function wildcards, and queries an intermediate classifier to provide a fast classification of the file, such as based on a predefined set of YARA rules (e.g., rules indicating the presence of a specific function in a file that indicate the file is malicious). System 200 may iteratively process the set of function codes extracted from the file until the earlier of the following: (i) an intermediate signature has been generated for each function code, and (ii) system 200 (e.g., intermediate classifier module 235) determines that the file can be classified based on the presence of one or more specific functions (e.g., based on performing a check against a set of YARA rules).

[0066] In some embodiments, system 200 includes an intermediate signature module 233. System 200 uses intermediate signature module 233 to generate an intermediate signature for a function. Intermediate signature module 233 computes the intermediate signature based on performing a hash function on wildcards of the function. The intermediate signature may also be referred to as a function hash. Examples of hash algorithms that can be combined with a deterministic signature include MD5, SHA-1, SHA-2, SHA-256, NTLM, LANMAN, RIPEMD-160, etc. Various other predefined procedures can be implemented to determine the intermediate signature.

[0067] In some embodiments, system 200 includes an intermediate classifier module 235. System 200 uses intermediate classifier module 235 to determine the classification of a file based at least in part on a subset of one or more intermediate signatures of the file. For example, if the file contains one or more specific functions, the system determines whether a subset of the intermediate signatures satisfies a YARA rule that indicates the file is malicious.

[0068] In some embodiments, system 200 includes a file signature determination module 237. The system uses the file signature determination module 237 to determine / generate a file signature for a file. The file signature determination module 237 determines the file signature based at least in part on all computed intermediate signatures. Alternatively, the file signature can be determined based at least in part on N intermediate signatures of file functions, where N is a positive integer less than the total number of functions associated with the file. In some embodiments, the file signature is performed at least in part based on performing a hash function on the intermediate signatures. Various hash functions can be implemented. For example, the hash function is the SSDeep hash function or the TLSH hash function. In some embodiments, the hash function used to compute the file signature differs from the hash function used to compute the various intermediate signatures.

[0069] In some embodiments, the file signature determination module 237 first sorts the intermediate signatures used to calculate the file signature, then concatenates the sorted intermediate signatures, and then calculates a hash for the concatenated sorted intermediate signatures. The intermediate signatures may be sorted based on values ​​or alphabetical order, etc.

[0070] In some embodiments, system 200 includes a classification module 239. System 200 uses classification module 239 to predict the classification of files, such as predicting whether a file is malicious or benign / non-malicious. For example, classification module 239 predicts the classification based at least in part on file signatures.

[0071] In response to determining representative information or identifiers (e.g., file signatures) associated with a .NET file, system 200 (e.g., classification module 239) may determine whether the .NET file corresponds to a previously analyzed .NET file (e.g., whether the .NET file matches historical information associated with previously calculated maliciousness determinations). For example, classification module 239 may query a database or a mapping of previously analyzed .NET files and / or historical information, such as a blacklist and / or whitelist of .NET files, in conjunction with determining whether the .NET file has been previously analyzed. In some embodiments, in response to determining that a .NET file does not correspond to a previously analyzed .NET file, classification module 239 uses a classifier (e.g., a model, such as a model trained using a machine learning process) to determine (e.g., predict) whether the .NET file is malicious. Additionally, or alternatively, system 200 may assign the .NET file to another system, module, or service responsible for analyzing the mismatched .NET files to generate classifications and / or corresponding YARA rules (if any). In some embodiments, in response to determining that the .NET file corresponds to a previously analyzed .NET file, the classification module 239 obtains an indication as to whether the corresponding previously analyzed .NET file is malicious. The classification module 239 may use the indication as to determine whether the received .NET file is malicious.

[0072] In some embodiments, system 200 includes a clustering module 241. System 200 uses clustering module 241 to implement clustering techniques by combining the identification of new sample clusters based on the degree of similarity of file signatures. For example, the system considers a group of samples to correspond to a specific cluster (e.g., a family of files, etc.) based on the determination that the degree of similarity between any two samples exceeds a predefined similarity threshold (e.g., 95% similarity or other predefined percentages). For example, clustering techniques include clustering algorithms such as K-means++ and X-means, used to check for new clusters and notify maintenance personnel (e.g., subject matter experts or another service for identifying urgent activities or tools) to conduct manual investigation and labeling. In response to the detection of a new cluster in a database of traffic samples, clustering module 241 can provide an indication of the new cluster.

[0073] In some embodiments, system 200 includes an allocation module 243. System 200 uses allocation module 243 to allocate file families (e.g., clusters of file signatures) to a service to review the clusters in conjunction with file classification, determination of YARA rules, or confirmation of predicted classifications obtained from classification module 239. This service may be another module, system, or running service that generates predicted classifications based at least in part on a query machine learning module. Additionally, or alternatively, this service may be a client associated with a human operator responsible for analyzing file families. In conjunction with file analysis, the system may generate YARA rules (if any), which are implemented to provide rapid classification, for example, through intermediate classifier module 235.

[0074] In some embodiments, system 200 includes a notification module 245. System 200 uses notification module 245 to provide indications of traffic classification, such as whether traffic is malicious, or whether traffic is malicious traffic. Additionally, or alternatively, system 200 uses the notification module to provide indications that traffic samples are attributed to known malicious exploits (e.g., activities or tools), or that traffic samples do not match known activities or tools. Notification module 245 provides indications (e.g., reports) to another system or service (e.g., an inline security or other security entity requesting traffic classification or otherwise processing of traffic), or to an administrator system (e.g., used by a network administrator when assessing security policy posture). Notification module 245 may also provide indications or recommendations for proactive measures to be implemented (e.g., recommendations for processing traffic based on traffic classification, recommendations for remedial actions to be performed in response to the detection of traffic samples matching known activities or tools, etc.).

[0075] System 200 may use notification module 245 to provide updates to traffic whitelists to one or more security entities (e.g., firewalls), nodes, or endpoints (e.g., client terminals), such as a whitelist of IP addresses (e.g., IP addresses that initiated HTTP requests) or a whitelist of traffic signatures. According to various embodiments, notification module 245 obtains hashes, signatures, or other unique identifiers associated with a domain (e.g., web pages of a domain) or network traffic, and combines these with hashes, signatures, or other unique identifiers associated with a sample (e.g., a file signature) to provide an indication of whether the sample is malicious.

[0076] If the signature of the received sample is included in the set of signatures of samples previously considered malicious (e.g., a blacklist of samples), the security entity or endpoint can block the transmission of the corresponding traffic, or block traffic to / from the client device from which it collects traffic.

[0077] In some embodiments, system 200 includes a security enforcement module 247. System 200 uses security enforcement module 247 to enforce one or more security policies regarding information such as network traffic, .NET files, and files. Security enforcement module 247 enforces one or more security policies based on whether it is determined that a .NET file is malicious. For example, in the case where system 200 is a security entity or a firewall, system 200 includes security enforcement module 247. Firewalls typically deny or allow network traffic based on sets of rules. These sets of rules are often referred to as policies (e.g., network policies, network security policies, security policies, etc.). For example, a firewall can filter inbound traffic by applying a set of rules or policies to prevent unwanted external traffic from reaching a protected device. A firewall can also filter outbound traffic by applying a set of rules or policies (e.g., allow, block, monitor, notify, or log, and / or may specify other actions in firewall rules or firewall policies that can be triggered based on various criteria, such as those described herein). A firewall can also filter local network (e.g., intranet) traffic by similarly applying a set of rules or policies. Other examples of policies include security policies that require scanning for threats in incoming (and / or outgoing) email attachments, website content, files exchanged through instant messaging programs, information obtained via web interfaces or other user interfaces (such as interfaces to database systems, such as SQL interfaces) and / or other file transfers.

[0078] According to various embodiments, storage device 215 includes one or more of file system data 260, code data 265, and / or prediction data 270. Storage device 215 includes shared memory (e.g., system network memory) and / or database data and / or user activity data.

[0079] In some embodiments, file system data 260 includes databases such as one or more datasets (e.g., one or more datasets for .NET files, exploit traffic and / or .NET file attributes, mappings of malicious indicators to .NET files or hashes, signatures or other unique identifiers of .NET files, mappings of indicators of whether a particular .NET file is benign, or hashes, signatures, or other unique identifiers of .NET files, etc.). File system data 260 includes data such as historical information related to .NET files (e.g., the maliciousness of .NET files), whitelists of .NET files considered safe (e.g., unsuspecting, benign, etc.), blacklists of .NET files considered suspicious or malicious (e.g., .NET files considered to be malicious with a probability exceeding a predetermined / preset probability threshold), and information associated with suspicious or malicious .NET files.

[0080] Code data 265 includes data related to the code of a file. For example, code data 265 includes files, disassembled binary files, MSIL, function or method code, etc.

[0081] Predictive data 270 includes information related to the determination of whether a .NET file analyzed by system 200 is malicious. For example, predictive data 270 stores indications that a .NET file is malicious, indications that a .NET file is benign, etc. This information may be obtained and provided by notification module 245 (e.g., transmitted to applicable security entities, endpoints, or other systems). In some embodiments, predictive data 270 includes a hash or signature of a .NET file (such as one analyzed by system 200 to determine whether such a .NET file is malicious), or includes historical datasets that have previously been assessed for malice, for example, by a third party. Predictive data 270 may include a mapping of hash values ​​to malice indications (e.g., indications corresponding to malicious or benign actions, etc.).

[0082] According to various embodiments, memory 220 includes application execution data 275. Application execution data 275 includes data obtained or used in conjunction with the execution of an application, such as an application that executes a hash function, an application that extracts information from a .NET file, and an application that extracts information from a file. In embodiments, the application includes one or more applications that perform one or more of the following: receiving and / or executing queries or tasks, generating reports and / or configuring information in response to executed queries or tasks and / or providing information to a user in response to queries or tasks. Other applications include any other suitable applications (e.g., index maintenance applications, communication applications, machine learning model applications, applications for detecting suspicious files, document preparation applications, report preparation applications, user interface applications, data analysis applications, anomaly detection applications, user authentication applications, security policy management / update applications, etc.).

[0083] Figure 3A This is the code description of a method contained in a benign .NET file example. In the example shown, Code 300 includes the code for a function / method implemented by a benign file. Code 300 is obtained by disassembling a file, such as a file obtained from a network via network communication (e.g., a file obtained by a security entity). Code 300 includes a set of operators and a corresponding set of operands.

[0084] Figure 3BThis is a schematic diagram of a wildcarding method based on the method code of methods contained in a benign .NET file example. In the example shown, wildcarded method 310 corresponds to the wildcarded version of the function / method corresponding to code 300. As shown, the wildcarded version of the function / method is independent of the data. The wildcarded version of the function / method is a representation of various operators in the function / method. For example, the system obtains code 300 by disassembling the file, identifies the functions / methods in the disassembly, and wildcards the functions / methods. For example, the system parses a .NET binary file and identifies methods, transforming the code of each MSIL code into a uniform format.

[0085] Figure 3C This is the code description of a method contained in a benign .NET file example. In the example shown, Code 320 includes the code for a function / method implemented by a benign file. Code 320 is obtained by disassembling a file, such as a file obtained from a network across network communication (e.g., a file obtained by a security entity). Code 320 includes a set of operators and a corresponding set of operands.

[0086] Figure 3D This is a diagram illustrating the wildcarding method, which is based on the method code of a method included in a benign .NET file example. In the example shown, wildcarded method 330 corresponds to the wildcarded version of the function / method corresponding to code 320.

[0087] Figure 4A This is the code description of a method contained in a malicious .NET file example. In the example shown, code 400 includes code for a function / method implemented by the malicious file. Code 400 is obtained by disassembling a file, such as a file obtained from a network across network communication (e.g., a file obtained by a secure entity). Code 400 includes a set of operators and a corresponding set of operands.

[0088] Figure 4B This is a schematic diagram of a wildcard-based method, which is based on the method code of a method contained in a malicious .NET file example. In the example shown, wildcard-based method 410 corresponds to the wildcard-based version of the function / method corresponding to code 400.

[0089] Figure 4CThis is the code description of a method contained in a malicious .NET file example. In the example shown, code 430 includes code for a function / method implemented by the malicious file. Code 430 is obtained based on disassembling a file, such as a file obtained from a network across network communication (e.g., a file obtained by a secure entity). Code 400 includes a set of operators and a corresponding set of operands.

[0090] Figure 4D This is a schematic diagram of a wildcard-based method, which is based on the method code of a method contained in a malicious .NET file example. In the example shown, wildcard-based method 440 corresponds to the wildcard-based version of the function / method corresponding to code 430.

[0091] Figure 5 This is a flowchart of a method for identifying file families based on file signatures according to various embodiments. In some embodiments, process 500 is at least partially constituted by Figure 1 System 100 and / or Figure 2 The system 200 implementation.

[0092] At point 505, the system obtains a set of one or more samples. These samples may correspond to files that are benign, malicious, grayware, or have an unknown classification (e.g., the maliciousness of the file is unknown). In some embodiments, the sample is a .NET file. The system combines determining file families (e.g., classifying samples into a set of families) and / or performing file classification (e.g., classifying files as benign, malicious, unknown, etc.) to determine the set of one or more samples to analyze.

[0093] At point 510, the system performs code analysis on a collection of one or more samples (e.g., files). For example, for each sample to be analyzed, the system parses the file binary (e.g., a .NET binary for a .NET file) and disassembles the sample's instructions (e.g., functions and other instructions implemented during the execution of the file code). The system can use tools such as dnfile and dncil to parse the file binary to obtain a disassembled version of the sample (e.g., the file's Microsoft Intermediate Language (MSIL)). MSIL code is typically extracted from the data segment of a PE executable.

[0094] After obtaining a disassembled version of the sample, the system identifies the methods in the code. MSIL code contains a list of instructions to be interpreted (e.g., operator information and operand information). In response to identifying / determining the methods in the sample's code (e.g., MSIL code), the system obtains a set of method codes corresponding to each method in the code, selects a method code (e.g., method MSIL code) from this set, and transforms this method code into a uniform format. In some embodiments, the system transforms the method code of each method in the sample code into a corresponding representation in a uniform format. In some embodiments, the system transforms the method codes of N methods in the sample code into corresponding representations in a uniform format, where N is a positive integer less than the total number of methods in the sample code. N can be predefined, for example, based on the analysis of multiple methods to be analyzed, so that the classifier has sufficient information to generate an accurate classification of the sample (e.g., classifying the maliciousness of the sample).

[0095] In some embodiments, the uniform format is independent of the data. For example, moving a register from A to B is represented as moving a register from C to D. A uniform format can be predefined, or a process can be used to transform method code into a uniform format. The system can recognize operators and operands in the method code and transform the method code in a way that represents operators independent of the specific operands implemented in the code. In some embodiments, the uniform format includes operators and predefined special characters to represent operands. For example, operands are replaced with special characters. Figure 3A As an example, entry 20E9 of code 300 shows that the system determines the operator and operand information as "07 00 00 04" represented by "7b". As shown in wildcarding method 310, the operand information is replaced with the special character "?". Various other special characters can be implemented. In some embodiments, the operand information is represented by a predefined length / number of special characters. In various other embodiments, each value / number in the operand information is replaced by a predefined special character, so the cumulative length of the operator and operands remains unchanged. The transformation of method code to a uniform format generates a wildcarded method that maintains the functionality of the method regardless of the specific register or memory address used.

[0096] In response to obtaining a wildcard-based method (e.g., after transforming the method code into a uniform format), the system classifies samples and / or performs family correlation based on the wildcard-based method to assess the correlation between a particular sample and another sample (e.g., a database of previously analyzed / classified samples).

[0097] At point 515, in conjunction with the classification / association of samples, the system generates file signatures at least in part based on wildcard-based methods. The system can generate file signatures based on all wildcard-based methods obtained for the samples. Alternatively, the system can generate file signatures based on N wildcard-based methods, where N is a predetermined positive integer less than the total number of wildcard-based methods obtained for the samples.

[0098] Additionally or alternatively, the system determines a file family or file classification based at least in part on a set of methods for determining wildcardization of a file that includes a subset of one or more wildcardization methods previously identified as malicious. For example, if a malicious family type always has a specific set of one or more methods, the system can define a YARA rule indicating that a sample is considered to belong to a malicious family type (or be malicious) when a particular sample has a specific set of one or more methods for a malicious family type. In response to the set of methods for determining wildcardization of a sample including a subset of wildcardization methods that match a specific set of one or more methods indicating that the sample is a malicious file type, the system can thus quickly determine that the sample corresponds to a malicious family type. The system can perform a check as it processes each wildcard method whether the wildcardization method indicates a correlation with a specific family type, potentially saving processing time for pending wildcard methods when it detects that the current wildcard method (or the current subset of processed wildcardization methods) matches a YARA rule indicating that the sample is malicious or a specific malicious family type.

[0099] In some embodiments, the system generates file signatures and / or classifies / associates samples based at least in part on one or more intermediate signatures (e.g., intermediate hashes) of a wildcard-based method implemented by the sample. The file signature may also be referred to herein as a DNCodeHash. For each wildcard-based method associated with the sample, the system determines the intermediate signature, for example, by performing a predefined hash function on the wildcard-based method. The intermediate signature can be used to perform lookups against a YARA rule set to quickly determine whether a sample can be considered malicious or belong to a specific family type based on the intermediate signature rather than the file signature (which is computationally more expensive to obtain). In some embodiments, the predefined hash function used to obtain the intermediate signature is an MD5 hash function.

[0100] In response to determining a set of intermediate signatures for a sample, the system generates a file signature based at least in part on that set of intermediate signatures for the sample. The set of intermediate signatures may include the intermediate signature of each wildcard-coded method, or may include N intermediate signatures of N wildcard-coded methods (e.g., the first N wildcard methods computed or invoked by the sample). In some embodiments, generating a file signature includes concatenating the intermediate signatures from the set of intermediate signatures and calculating the file signature based on the concatenated intermediate signature.

[0101] The concatenation of intermediate signatures can follow a predetermined process. For example, the system can first sort the set of intermediate signatures (e.g., based on individual intermediate hash values, such as alphabetical order), and then concatenate the sorted set of intermediate signatures. This sorting of intermediate signatures can provide a uniform representation even when different malicious files of a specific set of malicious family types execute methods in different orders.

[0102] In some embodiments, calculating a file signature based on the set of intermediate signatures (e.g., concatenated intermediate signatures) involves performing a predefined hash function on the set of intermediate signatures. The predefined hash function used to calculate the file signature can be a fuzzy hash function, such as SSDeep or TLSH.

[0103] In some embodiments, the system classifies samples at least in part based on performing a lookup on an index or dataset of samples to determine whether the file signature of a sample matches a known sample (e.g., a previously analyzed / classified sample). For example, the system may check a whitelist of benign / non-malicious samples to determine if the whitelist has an entry that matches the file signature of the sample being analyzed. In response to determining that the whitelist has a matching entry, the system may consider the sample to be benign / non-malicious. As another example, the system may check a blacklist of malicious samples to determine if the blacklist has an entry that matches the file signature of the sample being analyzed. In response to determining that the blacklist has a matching entry, the system may consider the sample to be malicious. Matching file signatures with signatures stored in an index or dataset can also be used to determine the family type to which a sample belongs (e.g., the index may be a signature-to-family-type mapping, etc.).

[0104] In some embodiments, the system performs clustering on a set of file signatures for a sample set. The system may consider file signatures with a similarity greater than a predefined similarity threshold (e.g., 95% similarity) as a single cluster. Furthermore, the system may consider clusters as corresponding family types. Clustering file signatures into a set of clusters allows for more efficient sample analysis. For example, the amount of samples processed may be very large, making the analysis of all samples expensive. Security services may receive hundreds or thousands of false negatives per week. In some implementations, samples are assigned to subject matter experts for manual review of the samples / family types. To avoid assigning two different samples belonging to the same family to two different human reviewers (which duplicates the work of reviewing a specific family type), the system may determine the corresponding cluster for that family type and assign that family type to a specific human reviewer (e.g., to assess whether a sample has been misclassified as benign / non-malicious).

[0105] At 520, the system identifies the family of the sample. For example, the system determines that the sample obtained at 505 can be divided into file family A 520, file family B 522, and benign file family 524. The system can then process the sample based on the family type to which it belongs. For example, samples of a family type for which there is no matching entry in the previously analyzed sample index / dataset can be handed over to a human reviewer or other service agency for classification (e.g., using a machine learning model to perform classification). In some embodiments, the system assigns samples / sample clusters to those samples that the classifier did not classify as malicious (e.g., unknown or benign / non-malicious classification samples, which may result in false negatives).

[0106] Figure 6 This is a flowchart of a method for classifying samples based at least in part on the file signatures of the samples according to various embodiments. In some embodiments, process 600 is at least partially... Figure 1 System 100 and / or Figure 2 The system 200 implementation.

[0107] At point 605, the system obtains the sample to be analyzed. The sample can be obtained from a security entity, such as a firewall, which acquires the sample based on communication across the corresponding enterprise network. At point 610, the system disassembles the sample to obtain its code (e.g., MSIL). The code is typically a direct representation of executable functionality, thus serving as an ideal basis for classifying the sample.

[0108] At point 615, the system performs function extraction. For example, the system identifies different functions / methods contained in the code (e.g., MSIL code). Each function / method may include a set of operators and their corresponding operands.

[0109] At 620, the system obtains the function wildcard (e.g., the method referred to herein as wildcardization). The function wildcard corresponds to a data-independent representation of the function, such as by generating an identifier that includes a set of operators and a set of predefined special characters to replace the function's operands.

[0110] At position 625, the system obtains a set of function hashes for the sample (e.g., also referred to as intermediate hashes in this paper). Function hashes can be generated based on executing predefined hash functions against function wildcards. For example, the system performs an MD5 hash of a function wildcard.

[0111] At point 630, the system uses function hashes to classify samples (e.g., predict whether a sample is malicious or non-malicious / benign) or clusters a sample with a group of other samples deemed to have similarity exceeding a predetermined similarity threshold. The system can generate a file signature based on a set of function hashes for the samples, such as by sorting the set of function hashes (e.g., by value or alphabetical order), concatenating the sorted set of function hashes, and executing a hash function (e.g., SSDep, TLSH) on the concatenated set of function hashes.

[0112] In some embodiments, the system uses file signatures to perform lookups against known samples (e.g., a set of previously classified malicious samples and / or a set of previously classified benign / non-malicious samples). In the example shown, the system performs a lookup against index / database 632. In response to determining that a file signature matches the signature of a known sample, the system can classify the sample corresponding to the file signature based on the classification of the matching known sample. In response to determining that a file signature does not match any entry (e.g., file signature) in index / database 632, the system can store the file signature for clustering or further analysis (e.g., by a manual reviewer or a machine learning model).

[0113] In some embodiments, the system uses file signatures to perform clustering on file signatures of samples that do not match entries stored in index / database 632. The system can determine cluster sets that respectively include file signatures / samples with similarity satisfying a predetermined similarity threshold.

[0114] At point 635, the system provides sample classification (e.g., an indication of whether a sample is malicious or benign / non-malicious) and / or assigns samples to another service for review and evaluation. This other service can be a manual reviewer or a classifier implementing a machine learning model. This service reviews samples classified as benign / malicious or indeterminate and evaluates whether a sample is malicious (e.g., determining whether the sample's predicted classification is a false negative (e.g., benign) or a false positive (malicious)).

[0115] Figure 7 This is an example of transforming function code into a wildcard function according to various embodiments. In the example shown, the system obtains a sample (e.g., a file), disassembles the sample into MSIL code, and extracts function code 705. The system identifies a set of operators 710 and a set of operands 715 in the function code 705. The system generates a wildcard function 720 based at least in part on the set of operators 710. For example, the system replaces operands in the operand set 715 with special characters (e.g., "?") to normalize the function and make the function data-independent.

[0116] Figure 8This is a flowchart of a method for performing sample classification according to various embodiments. In some embodiments, process 800 is at least partially constituted by Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 800 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 800 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0117] At 805, the system receives a sample including a file. This sample can be received from network traffic. For example, a firewall can capture files being transferred across a network. In some embodiments, this file is a Windows .NET file.

[0118] At point 810, the system uses hashing techniques to generate file signatures based on code. The system can generate file signatures based on disassembled code within the file. For example, the system can generate file signatures based on the code of functions / methods implemented by the file.

[0119] In some embodiments, the system determines the file signature based at least in part on data-independent characteristics of the code used for the function / method. For example, the system uses a wildcard representation of the function / method. The wildcard representation may correspond to operands contained in the function / method code.

[0120] In some embodiments, the hashing technique is predefined. Examples of hashing techniques that can be implemented include the SSDeep hash function and the TLSH hash function. Various other hash functions can also be implemented.

[0121] At point 815, the system classifies samples based on file signatures used in the code. The system can compare file signatures to a database or index of known malware or known benign files, and classify the samples based on their match with known samples in the database. For example, if a file signature and a known sample in the database or index have a similarity greater than a predefined similarity threshold, the system classifies the obtained file according to the classification of known samples (e.g., if the file signature is considered similar to a known malicious sample, the system considers the analyzed file malicious).

[0122] At position 820, the system provides sample classification.

[0123] At 825, it is determined whether process 800 is complete. In some embodiments, process 800 is determined to be complete in response to determining that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, or an administrator instructs that process 800 be paused or stopped, etc. In response to determining that process 800 is complete, process 800 ends. In response to determining that process 800 is not complete, process 800 returns to 805.

[0124] Figure 9 This is a flowchart of a method for performing sample classification according to various embodiments. In some embodiments, process 900 is at least partially performed by... Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 900 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 900 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0125] At position 905, the system receives a sample including a .NET file.

[0126] At 910, the system parses and disassembles the .NET file to obtain the methods implemented by the .NET file. For example, the system identifies functions in the code of the .NET file.

[0127] At point 915, the system determines whether the sample is malicious.

[0128] In response to determining at 915 that the sample is malicious, process 900 proceeds to 920. At 920, the system provides an indication that the sample is malicious. Conversely, in response to determining at 915 that the sample is not malicious, process 900 proceeds to 925. In some embodiments, in response to determining that the sample is not malicious, the system provides an indication that the sample is not malicious. The system or another system or service that provides sample classification processes network traffic (e.g., analyzed samples) based on sample classification. For example, the system enforces security policies on samples based on sample classification (e.g., malicious or benign).

[0129] At 925, it is determined whether process 900 is complete. In some embodiments, process 900 is determined to be complete in response to determining that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, or an administrator instructs that process 900 be paused or stopped. In response to determining that process 900 is complete, process 900 ends. In response to determining that process 900 is not complete, process 900 returns to 905.

[0130] Figure 10 This is a flowchart of a method for determining an intermediate hash of a function within code for a sample, according to various embodiments. In some embodiments, process 1000 is at least partially constituted by Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1000 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on client systems, targeting traffic transmitted across or into / out of the network. In some implementations, process 1000 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0131] At point 1005, the system receives an instruction to generate a file signature. The system determines to analyze the methods (e.g., functions) contained in the sample's code based on the determination that a malicious classification will be performed on the sample. Process 1000 can be called by another system, service, or process in response to such system, service, or process determining to determine a malicious classification. For example, process 1000 is called to obtain intermediate hashes (e.g., a set of hashes corresponding to a set of methods / functions contained in the sample or file), which will then be used to classify the sample.

[0132] At 1010, the system selects a method. The system identifies the method / function in the code of the file (e.g., the sample being analyzed).

[0133] At position 1015, the system wildcards the method (e.g., to obtain function wildcards or wildcardized methods).

[0134] At position 1020, the system generates a hash of the wildcard-based method. The hash of the wildcard-based method can be considered an intermediate hash. For example, a set of intermediate hashes of a set of methods in a file can be used to determine the file signature of a sample.

[0135] At point 1025, the system determines whether to analyze another method. For example, the system determines whether to wildcard another method and hashes such wildcarded method. In some embodiments, the system determines whether to wildcard another method based at least in part on one or more of the following: (i) the file includes other methods / functions that have not yet been wildcarded, and (ii) the wildcarded method does not indicate that the file is malicious.

[0136] In some embodiments, in response to obtaining a wildcarded method / function or its corresponding hash, the system compares the wildcarded method with a predefined set of YARA rules. A YARA rule may correspond to a set of one or more predefined rules, and if the set of rules is satisfied, it indicates that the sample is malicious. YARA rules may include rules indicating that a file containing a specific function or method is considered malicious (e.g., rules generated based on determining that any file with the corresponding function is a malicious file). In response to the system determining that the wildcarded method or hash matches a specific function known to be malicious or known to be included in a malicious file, the system considers the file malicious and determines not to analyze further methods / functions of the file. For example, the system may quickly determine whether a file is malicious based on methods / functions in the file that match YARA rules, thus preserving the processing of all methods / functions of the file.

[0137] In response to determining that another method needs to be analyzed, process 1000 returns to 1010, and process 1000 iterates on 1010-1025 until the system determines that no further method / function needs to be analyzed. Conversely, in response to determining that no further method needs to be analyzed, process 1000 advances to 1030.

[0138] At 1030, the system provides a hash for wildcarding methods.

[0139] At 1035, it is determined whether process 1000 is complete. In some embodiments, process 1000 is determined to be complete in response to determining that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, or an administrator instructs that process 1000 be paused or stopped, etc. In response to determining that process 1000 is complete, process 1000 ends. In response to determining that process 1000 is not complete, process 1000 returns to 1005.

[0140] Figure 11 This is a flowchart of a method for detecting malicious samples, according to various embodiments, based at least in part on an intermediate hash of a function within the code used for the sample. In some embodiments, process 1100 is at least in part... Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1100 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 1100 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0141] At 1105, the system receives an instruction to analyze the maliciousness of the method. In some embodiments, process 1100 is combined with process 1025 of 1000 to determine whether to analyze additional method calls. For example, 1025 calls process 1100 to perform a quick check to determine whether the currently analyzed method / function indicates that the sample (e.g., a file) is malicious, thereby avoiding the system processing all functions / methods in the sample to obtain a maliciousness classification. Process 1100 is called to determine the file classification (e.g., maliciousness classification) based on a specific function / method and a set of YARA rules.

[0142] At position 1110, the system obtains the hash using the wildcard method. For example, the system obtains the hash computed at position 1020 of process 1000.

[0143] At point 1115, the system obtains the YARA rule set. The YARA rule set can be predefined and stored in security policies, databases, or other indexes. The YARA rule set can be determined by subject matter experts or automatically based on machine learning techniques used to identify rules that can be used to determine whether a file is malicious.

[0144] At 1120, the system compares the hash of the wildcard method with the YARA rule set. In some embodiments, the system obtains a blacklist or index of functions or methods whose presence in a file indicates that the file is malicious. The system can perform a query / lookup against the blacklist or index to determine whether the hash of the wildcard method matches any entry in the blacklist or index.

[0145] At 1125, the system determines whether a sample is malicious, at least in part, based on a wildcarding method and a comparison of the YARA rule set. In response to determining that the method / function indicates the sample is malicious (e.g., the sample includes a function / method indicating the sample is malicious), process 1100 proceeds to 1130, where the system provides an indication that the sample is malicious. Conversely, in response to determining that the method / function does not indicate the sample is malicious, process 1100 proceeds to 1135, where the system provides an indication that the method / function does not indicate the sample is malicious. In some embodiments, the indication provided at 1130 or 1135 may be provided to the system, service, or process that calls process 1100.

[0146] At 1140, it is determined whether process 1100 is complete. In some embodiments, process 1100 is determined to be complete in response to determining that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, no further methods / functions to analyze, or an administrator instructs that process 1100 be paused or stopped, etc. In response to determining that process 1100 is complete, process 1100 ends. In response to determining that process 1100 is not complete, process 1100 returns to 1105.

[0147] Figure 12 This is a flowchart of a method for obtaining a file signature for a sample, according to various embodiments. In some embodiments, process 1200 is at least partially performed by... Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1200 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 1200 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0148] At point 1205, the system receives an instruction to generate a file signature.

[0149] At position 1210, the system obtains a set of method hashes.

[0150] At position 1215, the system hashes and concatenates the group of methods.

[0151] At position 1220, the system generates the hash of the concatenated set of method hashes. For example, the system uses the hash of this set of method hashes (e.g., intermediate hashes) to obtain the file signature of the sample.

[0152] At 1225, the system provides a file signature. In some embodiments, the system provides a file signature to the system, service, or process that calls procedure 1200.

[0153] At 1230, it is determined whether process 1200 has completed. In some embodiments, process 1200 is determined to be complete in response to the determination that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, the system / service / process that called process 1200 has acknowledged receiving the file signature, no further file signatures to obtain, or an administrator instructs process 1200 to pause or stop, etc. In response to the determination that process 1200 has completed, process 1200 ends. In response to the determination that process 1200 has not completed, process 1200 returns to 1205.

[0154] Figure 13 This is a flowchart of a method for clustering files based on their respective file signatures to obtain a set of file families, according to various embodiments. In some embodiments, process 1300 is at least partially performed by... Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1300 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 1300 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0155] In some embodiments, process 1300 is invoked periodically at a predetermined frequency. For example, the system periodically executes process 1300 to identify new types of files, such as emerging malicious activity or malware. In some embodiments, process 1300 is invoked in response to a predetermined number of unclassified files / samples (e.g., samples that do not match any samples in an index or database of categorized samples, such as whitelists or blacklists or samples) exceeding a predetermined number of sample thresholds.

[0156] At point 1305, the system receives an instruction to cluster the files.

[0157] At position 1310, the system obtains a set of file signatures for a set of files.

[0158] At position 1315, the system performs clustering on the file signatures of this group.

[0159] At position 1320, the system selects clustering.

[0160] At position 1325, the system provides clustering.

[0161] At point 1330, the system determines whether to provide additional clusters. For example, the system determines whether to identify and / or assign another cluster, such as to a service or subject matter expert, for analysis of that cluster.

[0162] In response to determining that another cluster should be provided or otherwise analyzed, process 1300 returns to 1320, and process 1300 iterates on 1320-1330 until no further clusters are provided or analyzed. Conversely, in response to determining that no further clusters are provided, process 1300 proceeds to 1335.

[0163] At 1335, it is determined whether process 1300 has completed. In some embodiments, process 1300 is determined to be complete in response to determining that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, the system / service / process that called process 1300 has acknowledged receiving the clustering, no further clustering to execute, or an administrator instructs process 1300 to pause or stop, etc. In response to determining that process 1300 has completed, process 1300 ends. In response to determining that process 1300 has not completed, process 1300 returns to 1305.

[0164] Figure 14 This is a flowchart of a method for obtaining analysis results for a set of file families according to various embodiments. In some embodiments, process 1400 is at least partially constituted by Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1400 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 1400 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0165] At point 1405, the system receives instructions to analyze clustering.

[0166] At point 1410, the system selects clustering.

[0167] At point 1415, the system assigns the selected cluster to the service used to analyze the file family.

[0168] At 1420, the system provides an indication of the assignment of the selected cluster.

[0169] At point 1425, the system determines that another cluster should be assigned.

[0170] In response to determining that another cluster needs to be assigned and / or analyzed, process 1400 returns to 1410, and process 1400 iterates on 1410-1425 until no further clusters need to be assigned or analyzed. Conversely, in response to determining that no further clusters need to be assigned or analyzed, process 1400 proceeds to 1430.

[0171] At point 1430, the system obtains the analysis results for a set of file families.

[0172] At 1435, it is determined whether process 1400 is complete. In some embodiments, process 1400 is determined to be complete in response to determining that there are no further samples or traffic to analyze (e.g., no further traffic prediction is required), no further traffic to process, the system / service / process that called process 1400 has acknowledged receiving the clustering, no further file families to analyze, an administrator instructs to pause or stop process 1400, etc. In response to determining that process 1400 is complete, process 1400 ends. In response to determining that process 1400 is not complete, process 1400 returns to 1405.

[0173] Figure 15 This is a flowchart of a method for detecting malicious files in received traffic according to various embodiments. In some embodiments, process 1500 is at least partially performed by... Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1500 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 1500 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0174] At 1605, an indication that the received sample is malicious is received. In some embodiments, the system receives (i) an indication that the sample is malicious and (ii) a sample or a hash, signature, or other unique identifier associated with the sample. For example, the system may receive an indication that the sample is malicious from a service such as a security or malware service. The system may receive an indication that the sample is malicious from one or more servers.

[0175] According to various embodiments, the indication that a sample is malicious is received in conjunction with an update to a set of previously identified malicious files. For example, the system receives an indication that a sample is malicious as an update to a blacklist of malicious files.

[0176] At 1510, the association between the stored sample and an indication that the sample is malicious is specified. In response to receiving an indication that the sample is malicious, the system stores the indication of maliciousness in association with the sample or an identifier corresponding to the sample to facilitate a lookup (e.g., a local lookup) of whether a subsequently received file is malicious. In some embodiments, the identifier corresponding to the sample stored in association with the indication that the sample is malicious includes a hash of the file (or a portion of a file), a signature of the file (or a portion of a file), or another unique identifier associated with the file. In some embodiments, storing the sample in association with the indication that the sample is malicious includes storing the signature of a .NET file in association with the indication that the sample is malicious. The signature of the .NET file may be a DNCodeHash (e.g., a hash calculated based on functions / methods in the file's code).

[0177] At position 1515, traffic is received. The system can receive traffic, such as traffic related to routing within / across the network, or redirect traffic into / out of the network, such as through a firewall, or monitor email or instant messaging traffic.

[0178] At 1520, a determination is made as to whether the traffic contains a malicious file. In some embodiments, the system obtains a file from the received traffic. For example, the system identifies the file as an email attachment, identifies the file as being exchanged between two client devices via an instant messaging program or other file exchange program, and so on. In response to obtaining the file from the traffic, the system determines whether the file corresponds to a file included in a set of previously identified malicious files (such as a blacklist of malicious files). In response to determining that the file is included in the set of files on the blacklist of malicious files, the system determines that the file is malicious (e.g., the system may further determine that the traffic contains a malicious file).

[0179] In some embodiments, the system determines whether the file corresponds to a file included in a set of previously identified benign files, such as a whitelist of benign files. In response to determining that the file is included in the set of files on the whitelist of benign files, the system determines that the file is not malicious (e.g., the system may further determine that the traffic includes malicious files).

[0180] According to various embodiments, in response to determining that the file is not included in a previously identified set of malicious files (e.g., a blacklist of malicious files) or is included in a previously identified set of benign files (e.g., a whitelist of benign files), the system considers the file to be non-malicious (e.g., benign).

[0181] According to various embodiments, in response to determining that a file is not included in a set of previously identified malicious files (e.g., a blacklist of malicious files) or a set of previously identified benign files (e.g., a whitelist of benign files), the system queries a malicious file detector (e.g., a machine learning classifier) ​​to determine whether the file is malicious. For example, the system may isolate the file until it receives a response from the malicious file detector regarding whether the file is malicious. The malicious file detector may perform the assessment of whether the file is malicious, such as simultaneously with the system's processing of traffic (e.g., in real time with queries from the system). The malicious file detector may correspond to... Figure 1 System 100 malicious file detector 170 and / or Figure 2 System 200.

[0182] In some embodiments, the system determines whether a file is included in a set of previously identified malicious files or a set of previously identified benign files by calculating a hash or determining a signature or other unique identifier associated with the file, and performing a lookup within a set of previously identified malicious files or a set of previously identified benign files to find a file that matches the hash, signature, or other unique identifier. Various hashing techniques can be implemented. According to various embodiments, determining whether a file is included in a set of previously identified malicious files or a set of previously identified benign files includes determining a signature (e.g., DNCodeHash) corresponding to the file, and determining whether the signature is included in a historical dataset (e.g., a dataset that includes the results of previous maliciousness determinations).

[0183] In response to the determination at 1520 that the traffic does not contain malicious files, process 1500 proceeds to 1530, where the file is treated as non-malicious traffic / information.

[0184] In response to determining at 1520 that the traffic does not contain malicious files, process 1500 proceeds to 1525, where the file is treated as malicious traffic / information. The system may process malicious traffic / information based at least in part on one or more policies such as one or more security policies.

[0185] According to various embodiments, treating files as malicious traffic / information may include performing proactive measures. These proactive measures may be performed based on (e.g., at least in part on) one or more security policies. For example, one or more security policies may be pre-configured by a network administrator, a customer (e.g., an organization / company) to provide a service for detecting malicious files, etc. Examples of proactive measures that may be performed include: separating files (e.g., isolating files), deleting files, alerting users to the detection of malicious files, providing a prompt to the user when a device attempts to open or execute a file, blocking file transfer, updating a blacklist of malicious files (e.g., mapping the file's hash to an indication that the file is malicious), etc.

[0186] At 1535, it is determined whether process 1500 is complete. In some embodiments, process 1500 is determined to be complete in response to determining that no further samples need to be analyzed (e.g., no further prediction of the documents is required), an administrator instructing pause or stop process 1500, etc. In response to determining that process 1500 is complete, process 1500 ends. In response to determining that process 1500 is not complete, process 1500 returns to 1505.

[0187] Figure 16 This is a flowchart of a method for classifying and processing files received in network traffic, according to various embodiments. In some embodiments, process 1600 is at least partially... Figure 1 System 100 and / or Figure 2 The system 200 is implemented. In some implementations, process 1600 can be implemented by a security entity (e.g., a firewall), such as by implementing security policies in conjunction with anti-malware applications running on the client system, targeting traffic transmitted across or into / out of the network. In some implementations, process 1600 can be implemented by a client device such as a laptop computer, smartphone, or personal computer.

[0188] At point 1605, the system obtains the file from the traffic. The system may obtain the traffic, for example, by combining traffic routed within or across a network, or by passing traffic within / out of a network such as a firewall, or by monitoring email or instant messaging traffic. In some embodiments, the system obtains the file from the received traffic. For example, the system may identify the file as an email attachment, identify the file as being exchanged between two client devices via an instant messaging program or other file exchange program, and so on.

[0189] At 1610, the system determines a signature corresponding to the file. In some embodiments, the system calculates a hash or determines a signature or other unique identifier associated with the file, such as a hash calculated for the function / method code in the file. Various hash techniques can be implemented. For example, the hash technique can determine (e.g., calculate) the MD5 hash of the file. As another example, the hash technique can be based on the SSDeep hash function to determine (e.g., calculate) the hash.

[0190] In some embodiments, determining a signature corresponding to a file includes calculating the DNCodeHash of the file (e.g., a.NET file). In some embodiments, the system disassembles the file, identifies functions / methods in the code, wildcardizes the functions / methods (e.g., determines the operands and ignores the operands), calculates the hash of the wildcardized functions / methods (e.g., determines an intermediate hash, such as an MD5 hash), concatenates the hashes of the wildcardized functions / methods (e.g., the system sorts the hashes of the wildcardized functions / methods and then concatenates the sorted hashes), and calculates the hash of the concatenated sorted hashes of the wildcardized functions / methods.

[0191] At 1615, the system queries a dataset of signatures of malicious files to determine whether the signature corresponding to the file matches the signature from a malicious file. In some embodiments, the system looks up in the dataset the signature of a malicious file of a file that matches a hash, signature, or other unique identifier. The dataset of malicious file signatures can be stored locally at the system or remotely on a storage system accessible to the system.

[0192] According to various embodiments, determining whether the file is included in a set of previously identified malicious files or a set of previously identified benign files includes determining a file signature (e.g., DNCodeHash) corresponding to the file, and determining whether the file signature (e.g., DNCodeHash) is included in a historical dataset (e.g., a dataset including the results of previous maliciousness determinations).

[0193] At 1620, the system determines whether the file is malicious based at least in part on whether the file signature of the file matches the signature of a malicious sample. In some embodiments, the system determines whether the dataset of malicious signatures includes a record that matches the file signature of the file obtained from the traffic. In response to determining that the historical dataset includes an indication that a file corresponding to the file signature (e.g., DNCodeHash) is malicious (e.g., the DNCodeHash is included in a blacklist of fields), the system considers the file obtained from the traffic at 1605 to be malicious.

[0194] At point 1625, the file is processed based on whether it is malicious. In some embodiments, in response to determining that the file is malicious, the system applies one or more security policies to the file. In some embodiments, in response to determining that the file is not malicious, the system treats the file as benign (e.g., treats the file as normal traffic).

[0195] At 1630, it is determined whether process 1600 is complete. In some embodiments, process 1600 is determined to be complete in response to determining that no further samples need to be analyzed (e.g., no further prediction of the documents is required), an administrator instructing pause or stop process 1600, etc. In response to determining that process 1600 is complete, process 1600 ends. In response to determining that process 1600 is not complete, process 1600 returns to 1605.

[0196] Various examples of the embodiments described herein are illustrated with reference to flowcharts. Although the examples may include certain steps performed in a particular order, various steps may be performed in various orders according to various embodiments, and / or various steps may be combined into a single step or performed in parallel.

[0197] Although the embodiments described above have been given in detail for clarity of understanding, the invention is not limited to the details provided. Many alternative methods of carrying out the invention are available. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A system for generating signatures for Windows .NET binaries, comprising: One or more processors, which are configured as follows: Use hashing techniques to generate file signatures based on code; and The samples are classified based on the code using the file signature; and A memory coupled to the one or more processors and configured to provide instructions to the one or more processors.

2. The system according to claim 1, wherein the code corresponds to a Windows .NET binary file.

3. The system according to claim 1, wherein the hashing technique is the MD5 hash function.

4. The system according to claim 1, wherein the hashing technique is the SSDeep hash function.

5. The system according to claim 1, wherein the hashing technique is a TLSH hash function.

6. The system of claim 1, wherein classifying the samples based on the code using the file signature comprises: The sample is determined to be malicious based at least in part on the file signature.

7. The system of claim 1, wherein classifying the samples based on the code using the file signature comprises: The sample is determined to be benign based at least in part on the file signature.

8. The system of claim 1, wherein the one or more processors are further configured to: The samples are processed based at least in part on sample classification.

9. The system of claim 8, wherein the sample is processed based at least in part on a predefined security policy.

10. The system of claim 8, wherein processing the sample includes performing proactive measures in response to determining that the sample corresponds to a malicious sample.

11. The system of claim 1, wherein the one or more processors are further configured to parse the Windows .NET binary and disassemble the methods implemented by the Windows .NET binary into the code.

12. The system of claim 1, wherein the code corresponds to the code of one or more methods implemented by the Windows.NET binary.

13. The system according to claim 12, wherein: The code was obtained by parsing the Windows.NET binary file; The code of the one or more methods is transformed into one or more transformation results based on a uniform format; and One or more transformation results are hashed to obtain one or more hash results, and a file signature is generated based on the one or more hash results.

14. The system of claim 13, wherein the one or more hash results correspond to intermediate hashes, and a concatenated hash is obtained based on concatenating the intermediate hashes, and a final hash result is obtained by performing hashing on the concatenated hash.

15. The system of claim 14, wherein the hashing technique used to obtain the concatenated hash is the SSDeep hash function or the TLSH hash function.

16. The system of claim 13, wherein one or more transformation results use wildcards for operands in the corresponding method and are independent of the data.

17. The system of claim 16, wherein the one or more processors are further configured to: Based on the results of one or more transformations, Yara rules are generated to identify known malware function methods.

18. The system of claim 1, wherein the file signature is used in combination with one or more of malware learning, malware detection, and malware clustering.

19. The system of claim 1, wherein the signatures are clustered in response to determining that a set of signatures of a set of files has a similarity score higher than a predefined similarity threshold.

20. The system of claim 19, wherein the predefined similarity threshold is greater than 95%.

21. The system of claim 1, wherein a set of signatures of trusted or benign code samples is clustered in conjunction with a generated code whitelist.

22. The system of claim 1, wherein the sample is identified as malware based on a match between the file signature and the signature of a known malware.

23. The system of claim 1, wherein the sample is considered benign code based on the matching of the file signature with the signature of known benign code.

24. A method for generating a signature for a Windows .NET binary file, comprising: Use hashing techniques to generate file signatures based on code; and The file signature is used to classify the samples based on the code.

25. A computer program product embodied in a non-transitory computer-readable medium for generating a signature for a Windows .NET binary file, wherein the computer program product includes computer instructions for: Use hashing techniques to generate file signatures based on code; and The file signature is used to classify the samples based on the code.

Images