Automatic detection of unknown packers

Abstract

Techniques for automatically detecting unknown packers are disclosed. In some embodiments, a system / process / computer program product for automatically detecting unknown packers includes receiving a plurality of samples for malware packer detection analysis, executing a packer filter to determine whether each of the plurality of samples is packed, emulating each of the packed samples to extract a plurality of features, and clustering the packed samples based on the extracted features.

Description

Background Art

[0001] Malware is a common term commonly used to refer to malicious software (including, for example, various hostile, invasive, and / or other unwanted software). Malware can be in the form of code, scripts, active content, and / or other software. Examples of the use of malware include interrupting computer and / or network operations, stealing confidential information (such as information related to identity, finance, and / or intellectual property), and / or gaining access to private / dedicated computer systems and / or computer networks. Unfortunately, as techniques have been developed to help detect and mitigate malware, nefarious authors have found ways to avoid such efforts. Accordingly, there continues to be a need for improvement in techniques for identifying and mitigating malware.

Brief Description of the Drawings

[0002] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 6

Figure 7

[0003] The present invention can be implemented in a number of ways, including a process, an apparatus, a system, a composition, a computer program product embodied on a computer-readable storage medium, and / or instructions stored in a memory and / or executed by a processor coupled to the memory and / or provided by the memory. In this specification, these implementations, or any other form the present invention may take, may be referred to as a technique. Generally, the order of steps of the disclosed process may be changed within the scope of the present invention. Unless otherwise specified, components such as a processor or a memory described as being configured to perform a task are implemented as general-purpose components temporarily configured to perform the task at a given time or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0004] A detailed description of one or more embodiments of the present invention is provided below along with the accompanying drawings that illustrate the principles of the present invention. The present invention is described in relation to such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims and the present invention includes numerous alternatives, modifications, and equivalents. In order to provide a complete understanding of the present invention, numerous specific details are set forth in the following description. These details are provided for purposes of illustration and the present invention may be practiced according to the claims without some or all of these specific details. Technical material known in the technical fields associated with the present invention is not described in detail so as not to unnecessarily obscure the present invention.

[0005] A firewall generally enables authorized communications to pass through the firewall while protecting the network from unauthorized access. A firewall is typically a device, a set of devices, or software running on a device that provides firewall functionality for network access. For example, a firewall can be integrated into the operating system of a device (such as a computer, smartphone, or other type of network - communicable device). A firewall can also be integrated into or otherwise run within one or more software applications on various types of devices such as computer servers, gateways, network / routing devices (such as network routers), and data appliances (such as security appliances or other types of dedicated devices). And in various implementations, certain operations can be implemented within dedicated hardware such as an ASIC or FPGA.

[0006] A firewall typically rejects or permits network transmissions based on a set of rules. This set of rules is often referred to as a policy (e.g., a network policy or a network security policy). For example, a firewall can filter inbound traffic by applying a set of rules or policies to prevent unwanted external traffic from reaching a protected device. A firewall can also filter outbound traffic by applying a set of rules or policies (e.g., allow, block, monitor, notify, or log, and / or other actions that can be specified in a firewall rule or firewall policy, which can be triggered based on various criteria as described herein). A firewall can also filter local network (e.g., intranet) traffic by applying a set of rules or policies in a similar manner.

[0007] A security device (e.g., a security appliance, a security gateway, a security service, and / or other security devices) can perform various security operations (e.g., firewall, anti-malware, intrusion prevention / detection, proxy, and / or other security functions), network functions (e.g., routing, quality of service (QoS), workload balancing of network-related resources, and / or other network functions), and / or other security and / or network-related functions. For example, a routing function can be based on source information (e.g., IP address and port), destination information (e.g., IP address and port), and protocol information.

[0008] A basic packet filtering firewall filters network communication traffic by inspecting individual packets transmitted over the network (e.g., a packet filtering firewall or first-generation firewall, such as a stateless packet filtering firewall). A stateless packet filtering firewall typically inspects each individual packet itself and applies rules based on the inspected packet (e.g., using a combination of source and destination address information, protocol information, and port numbers of the packet).

[0009] An application firewall can also perform application layer filtering (e.g., an application layer filtering firewall or second-generation firewall that functions at the application level of the TCP / IP stack). An application layer filtering firewall or application firewall can generally identify a given application and protocol (e.g., web browsing using the Hypertext Transfer Protocol (HTTP), Domain Name System (DNS) requests, file transfers using the File Transfer Protocol (FTP), and various other types of applications and other protocols such as Telnet, DHCP, TCP, UDP, and TFTP (GSS)). For example, an application firewall can block unauthorized protocols that attempt to communicate on standard ports (e.g., unauthorized / rogue protocols that attempt to sneak through by using non-standard ports for that protocol can generally be identified using an application firewall).

[0010] A stateful firewall can also perform state-based packet inspection, where each packet is inspected within a set of packet contexts associated with the packet flow of its network transmission. This firewall technique is commonly referred to as stateful packet inspection. This is because it keeps track of all connections passing through the firewall and can determine whether a packet is the start of a new connection, part of an existing connection, or an invalid packet. For example, the state of a connection can itself be one of the criteria that triggers a rule in the policy.

[0011] As described above, advanced or next-generation firewalls can perform stateless and stateful packet filtering and application layer filtering. Next-generation firewalls can also perform additional firewall technologies. For example, a given new firewall, sometimes referred to as an advanced or next-generation firewall, can also identify users and content (e.g., next-generation firewall). In particular, a given next-generation firewall has expanded the list of applications that these firewalls can automatically identify to up to thousands of applications. Examples of such next-generation firewalls are commercially available from Palo Alto Networks (e.g., Palo Alto Networks' PA series firewalls). For example, Palo Alto Networks' next-generation firewalls use various identification technologies to enable enterprises to identify and control applications, users, and content - not just ports, IP addresses, and packets. The various identification technologies include Application ID (App-ID) for accurate application identification, User ID (User-ID) for user identification (e.g., by user or user group), and Content ID (Content-ID) for real-time content scanning (e.g., to control web surfing and restrict data and file transfers). These identification technologies enable enterprises to use business-related concepts to safely enable the use of applications, rather than following the traditional approach provided by conventional port-blocking firewalls.In addition, dedicated hardware for next-generation firewalls (e.g., implemented as a dedicated appliance) generally provides a higher performance level for application inspection than software running on general-purpose hardware (e.g., security appliances provided by Palo Alto Networks, which use dedicated, function-specific processing tightly integrated with a single-pass software engine to minimize latency while maximizing network throughput).

[0012] Advanced or next-generation firewalls can also be implemented using virtualized firewalls. Examples of such next-generation firewalls are commercially available from Palo Alto Networks (e.g., VMware(R) ESXi TM and NSX TM , Citrix(R) Netscaler SDX TM , the Palo Alto Networks VM Series firewalls that support various commercial virtualization environments including KVM / OpenStack (Centos / RHEL, Ubuntu(R)), and Amazon Web Services (AWS), as well as the CN Series container next-generation firewalls. For example, virtualized firewalls can support similar or identical next-generation firewall and advanced threat prevention capabilities available in physical form factor appliances, enabling enterprises to safely allow applications to flow into and across their private, public, and hybrid cloud computing environments. Through automation features such as VM monitoring, dynamic address groups, and REST-based APIs, enterprises can proactively monitor VM changes that supply their context into security policies, thereby removing policy silos that can occur when VMs are changed.

[0013] Overview of Techniques for Automatically Detecting Unknown Packers

[0014] Packers are generally used by malware creators as a mechanism to avoid malware detection by anti-malware security solutions. The generation of malware variants is also facilitated by such automated packers.

[0015] However, existing anti-malware security solutions cannot efficiently and effectively detect wild unknown / custom packers. For example, existing signature-based (for further information on signature-based techniques such as creating YARA rules based on code, see, for example, https: / / www.binarydefense.com / creating-yara-rules-based-on-code / ), and machine learning implementation detection solutions generally cannot automatically detect such unknown and / or new / custom packers.

[0016] Therefore, what is needed is an anti-malware security solution that can efficiently and automatically detect such unknown and / or new / custom packers.

[0017] Therefore, novel and improved techniques for automatically detecting unknown packers are disclosed.

[0018] In some embodiments, a system / process / computer program product for automatically detecting an unknown packer includes receiving a plurality of samples for malware packer detection analysis, performing a packer filter to determine whether each of the plurality of samples is packed, emulating each of the packed samples, extracting a plurality of features, and clustering the packed samples based on the extracted features.

[0019] For example, the plurality of features can include at least one or more of the following. That is, opcode features, file format features, and application programming interface (API) features, as further described below with respect to various embodiments.

[0020] In some embodiments, a system / process / computer program product for automatically detecting an unknown packer includes receiving a sample for inline malware packer detection analysis, performing a packer filter to determine whether the sample is packed, comparing with known packer clusters to determine whether the sample is associated with a known malware packer, and performing a response action based on the determination of the malware packer.

[0021] For example, the disclosed techniques provide a new approach to the technically difficult packer problem for malware detection. Specifically, the disclosed techniques are based on the observation that at the beginning of the entry point, the same packer stub contains a similar concept / flow of the unpacking routine, as further described below, to provide an automated solution for discovering and identifying such new and / or custom packers (e.g., packers can be identified as similar and grouped based on their unpacking flow and API calls).

[0022] More specifically, as further described below, for various embodiments, a new packer filter is disclosed that can automatically and effectively detect packed samples (e.g., malware samples). Also, machine learning, domain knowledge, and emulators can be used to automatically identify and group packers having similar unpacking routines. Test results demonstrate that the disclosed techniques can accurately identify new unknown packers that other approaches cannot identify. Further, the disclosed techniques are resistant to random / junk instructions (e.g., inserting random / junk instructions into such packers is often used as a pattern avoidance mechanism by malware writers) that would otherwise avoid detection by existing, conventional signature approaches, as further described below.

[0023] The techniques disclosed for automatically detecting unknown packers reduce the labor-intensive effort required by malware analysts to manually analyze packers in an attempt to detect and generate new signatures for previously unknown packers, as compared to existing, conventional signature approaches. Conventionally, malware analysts have had to find such new / custom packers, group / classify the packers, and manually create new signatures for the packers.

[0024] Also, as similarly described above, the techniques disclosed for automatically detecting unknown packers are generally superior to existing, conventional signature detection approaches. This is because signature-based detection approaches are circumvented by malware writers using junk instruction / pattern avoidance approaches that can avoid detection by such signatures. Further, such junk instruction / pattern avoidance approaches can result in hash changes that may require additional / re-analysis of previously identified packers for known malware, which is costly for security solutions.

[0025] Furthermore, the correlation of packer information, as further described below, can extend ground truth beyond multi-scanning and VirusTotal (e.g., VirusTotal is an online free service established in 2004 that analyzes files and URLs for viruses, worms, Trojan horses, and other types of malicious content. See https: / / www.VirusTotal.com). For example, if samples are not available in VirusTotal but have the same custom unpacking routine associated with a malware family, they can be determined to be highly related.

[0026] Accordingly, a new and improved security solution that facilitates the automatic detection of unknown packers using a security platform (e.g., a firewall (FW) / next-generation firewall (NGFW), a network sensor operating in place of a firewall, or another (virtual) device / component that can implement security policies using the disclosed techniques, including, for example, Palo Alto Networks' PA series next-generation firewalls, Palo Alto Networks' VM series virtualized next-generation firewalls, and CN series container next-generation firewalls, and / or other commercially available virtual-based or container-based firewalls) can be implemented and configured in accordance with some embodiments.

[0027] These and other embodiments and examples for automatically detecting unknown packers are further described below.

[0028] Exemplary System Architecture for Automatically Detecting Unknown Packers

[0029] Accordingly, in some embodiments, the disclosed techniques include providing a security platform (e.g., a security function / platform that can implement security policies using the disclosed techniques such as a firewall (FW) / next-generation firewall (NGFW), a network sensor operating in place of a firewall, or another (virtual) device / component such as PANOS running on a commercially available virtual / physical NGFW solution from Palo Alto Networks, or another security platform / NFGW including, for example, Palo Alto Networks' PA series next-generation firewall, Palo Alto Networks' VM series virtualized next-generation firewall, and CN series container next-generation firewall, and / or other commercially available virtual-based or container-based firewalls that can be similarly implemented and configured to execute the disclosed techniques) configured to provide a DPI capability (e.g., including stateful inspection) that applies the disclosed techniques to automatically detect unknown packers, as further described below.

[0030] FIG. 1 shows one exemplary environment in which malicious applications (“malware”) are detected and prevented from causing harm. As described in more detail below, malware classification can be shared and / or refined among various entities included in the environment shown in FIG. 1 (e.g., as determined by security platform 122). And using the techniques described herein, devices such as endpoint client devices 104 - 110 can be protected from such malware (e.g., including previously unknown / custom-packer-related malware).

[0031] As used herein, "malware" refers to applications that engage in behavior that the user would not approve of / would not approve, whether or not it is secret (and whether or not it is illegal), when fully informed. Examples of malware include Trojan horses, viruses, rootkits, spyware, hacking tools, etc. One example of malware is a desktop / mobile application that encrypts the user's stored data (e.g., ransomware). Another example of malware is a desktop / mobile application that collects the end user's activities and / or various information associated with the user and reports it to a remote server (e.g., spyware). Other forms of malware (e.g., keyloggers) can also be detected / blocked using improved and automated packer detection techniques as further described herein.

[0032] The term "packer" is used throughout this specification and is also referred to as "runtime packer", which is also known as "self-extracting archives" collectively. Thus, a packer can be used to provide software that unpacks itself in memory when the "packed file" is executed. This approach is also sometimes referred to as "executable compression". This type of compression was originally developed to make files smaller for distribution and storage, but is now often used by malware creators (e.g., hackers) to mask malware (e.g., to avoid analysis and detection by existing security solutions).

[0033] Packers (e.g., malware packers) are now commonly used by malware authors (e.g., hackers) to avoid detection by existing security solutions (for more information on how malware packers use various tricks to avoid analysis and detection by existing security solutions, see, for example, https: / / www.mcafee.com / blogs / enterprise / malware-packers-use-tricks-avoid-analysis-detection / ). Specifically, a malware packer is a tool that can be used to mask malware (e.g., to conceal / obfuscate malicious files / payloads from detection by existing security solutions). More specifically, a packer can encrypt, compress, and / or simply change the format of a malware file payload to make it appear as some other content in order to avoid malware analysis and detection by existing security solutions.

[0034] There are many existing commercially available packers. Attackers / hackers can use such existing commercially available packers to pack their malware files / payloads, but hackers often modify or customize these packers to make it more technically difficult for security solutions to detect the packed malware.

[0035] Given that packers are commonly used by hackers, packer detection can be used by security solutions to detect malware, as well as to identify and classify / group malware based on associated packers.

[0036] However, as further described herein, the automatic detection of packer activity is becoming an increasingly technical challenge as new and / or custom packers become more prevalent and sophisticated, and as they often evade detection by existing packer detection techniques, including signature-based packer detection and static machine learning-related packer detection techniques.

[0037] The techniques described herein can be used for the automatic detection of various packers (e.g., new and / or custom packers, etc.) with and / or for various platforms (e.g., desktops, mobile devices, game platforms, embedded systems, etc.). In the exemplary environment shown in FIG. 1, client devices 104-108 are laptop computers, desktop computers, and tablets (respectively) that exist within enterprise network 140. Client device 110 is a laptop computer that exists outside enterprise network 140.

[0038] Data appliance 102 is configured to enforce policies regarding communications between client devices, such as client devices 104 and 106, and nodes external to enterprise network 140 (e.g., those reachable via external network 118). Examples of such policies include those that manage traffic shaping, quality of service, and traffic routing. Other examples of policies include security policies that require scanning for threats in incoming (and / or outgoing) email attachments, web site content, files exchanged via instant messaging programs, and / or other file transfers. In some embodiments, data appliance 102 is also configured to enforce policies regarding traffic that remains within enterprise network 140.

[0039] One embodiment of a data appliance is shown in FIG. 2A. The example shown is a representation of the physical components included in data appliance 102 in various embodiments. Specifically, data appliance 102 includes a high-performance multi-core central processing unit (CPU) 202 and random access memory (RAM) 204. Data appliance 102 also includes storage 210 (such as one or more hard disks or solid state units). In various embodiments, data appliance 102 is used to monitor enterprise network 110 and store information (in any of RAM 204, storage 210, and / or other appropriate locations) used to implement the disclosed techniques. Examples of such information include application identifiers, content identifiers, user identifiers, requested URLs, IP address mappings, policies and other configuration information, signatures, host name / URL classification information, malware profiles, machine learning (ML) models. Data appliance 102 may also include one or more optional hardware accelerators. For example, data appliance 102 may include an encryption engine 206 configured to perform encryption and decryption operations, and one or more field programmable gate arrays 208 configured to perform matching, operate as a network processor, and / or perform other tasks.

[0040] The functionality described herein as being performed by the data appliance 102 can be provided / implemented in a variety of ways. For example, the data appliance 102 can be a dedicated device or set of devices. The functionality provided by the data appliance 102 can also be integrated or executed as software on a general-purpose computer, computer server, gateway, and / or network / routing device. In some embodiments, at least some of the services described as being provided by the data appliance 102 are instead (or in addition to) provided to a client device (e.g., client device 104 or client device 110) by software running on a client device (e.g., the endpoint protection application 132).

[0041] Whenever the data appliance 102 is described as performing a task, a single component, a subset of components, or all components of the data appliance 102 can cooperate to perform the task. Similarly, whenever a component of the data appliance 102 is described as performing a task, a sub-component can perform the task and / or a component can perform the task with other components. In various embodiments, a portion of the data appliance 102 is provided by one or more third parties. Depending on factors such as the amount of computing resources available to the data appliance 102, various logical components and / or features of the data appliance 102 may be omitted, and the techniques described herein are adapted accordingly. Similarly, additional logical components / features can be included in embodiments of the data appliance 102 as applicable. One example of a component included in the data appliance 102 in various embodiments is an application identification engine configured to identify applications (e.g., using various application signatures to identify applications based on packet flow analysis). For example, the application identification engine can determine the type of traffic involved in a session. Such as web browsing - social networking, web browsing - news, SSH, etc.

[0042] Figure 2B is a functional diagram of the logical components according to one embodiment of a data appliance. The example shown is a representation of logical components that may be included in the data appliance 102 in various embodiments. Unless otherwise specified, the various logical components of the data appliance 102 can generally be implemented in various ways, including a set of one or more scripts (e.g., written in Java®, python, etc. when applicable).

[0043] As shown, data appliance 102 includes a firewall and includes a management plane 232 and a data plane 234. The management plane is responsible for managing user interaction, such as providing a user interface for policy setting and viewing log data. The data plane is responsible for data management, such as performing packet processing and session processing.

[0044] Network processor 236 is configured to receive packets from a client device, such as client device 108, and provide them to data plane 234 for processing. Flow module 238 generates a new session flow whenever it identifies a packet as part of a new session. Subsequent packets are identified as belonging to the session based on a flow lookup. If applicable, SSL decryption is applied by SSL decryption engine 240. Otherwise, processing by SSL decryption engine 240 is omitted. Decryption engine 240 helps the data appliance 102 inspect and control encrypted traffic for SSL / TLS and SSH, and thus helps stop threats that might otherwise remain hidden within the encrypted traffic. Decryption engine 240 can also help prevent highly confidential content from leaving the enterprise network 140. Decryption can be selectively controlled (e.g., enabled or disabled) based on parameters such as URL category, traffic source, traffic destination, user, user group, and port. In addition to a decryption policy (e.g., one that specifies sessions to decrypt), a decryption profile can be assigned to control various options for sessions controlled by the policy. For example, the use of a particular cipher suite and encryption protocol version can be required.

[0045] The Application Identification (APP-ID) Engine 242 is configured to determine the type of traffic a session is involved in. As one example, the Application Identification Engine 242 can recognize a GET request within received data and conclude that the session requires an HTTP decoder. In some cases, the identified application, such as a web browsing session, can change, and such changes are noted by the data appliance 102. For example, a user can first browse a corporate Wiki (classified as "Web Browsing-Productivity" based on the visited URL) and then browse a social networking site (classified as "Web Browsing-Social Networking" based on the visited URL). Different types of protocols have corresponding decoders.

[0046] Based on the decisions made by the Application Identification Engine 242, packets are sent by the Threat Engine 244 to an appropriate decoder configured to assemble the packets (which may be received out of order) in the correct order, perform tokenization, and extract information. The Threat Engine 244 also performs signature matching to determine what should happen to the packets. If necessary, the SSL Encryption Engine 246 can re-encrypt the decrypted data. The packets are forwarded for transfer (e.g., to a destination) using the Forwarding Module 248.

[0047] Also, as shown in Figure 2B, policy 252 is also received and stored in the management plane 232. The policy can include one or more rules that can be specified using a domain name and / or a host / server name, and the rules can apply one or more signatures or other matching criteria or discovery methods, such as for the implementation of security policies for subscribers / IP flows, based on various extracted parameters / information from the monitored session traffic flow. Exemplary policies can include a packer / malware detection policy that uses the disclosed techniques for the automatic detection of unknown / custom packers. An interface (I / F) communicator 250 is provided for management communication (e.g., via a (REST) API, message, or network protocol communication, or other communicator constructs).

[0048] Security Platform

[0049] Returning to Figure 1, assume that a malicious individual creates malware 130 (using system 120) (e.g., when a user visits / browses a compromised website, malware can be delivered to the user's endpoint device via the compromised website or via a phishing attack, etc.). The malicious individual wants a client device, such as client device 104, to execute a copy of malware 130, unpack the malware executable file / payload, expose the client device to danger, and, for example, turn the client device into a bot within a botnet. The exposed client device is then instructed to execute a task (e.g., participate in a cryptocurrency mining or a denial of service attack) and report information to an external entity, such as command and control (C&C) server 150, and, if applicable, receive commands from C&C server 150.

[0050] Assume that data appliance 102 intercepts an email sent to user “Alice” who operates client device 104 (e.g., by system 120). In this example, Alice receives the email and clicks on a link to a phishing / risky site, which may result in an attempt by Alice's client device 104 to download malware 130. However, in this example, data appliance 102 performs the disclosed techniques for automatic detection of unknown / custom packers and blocks access to the packed malware content from Alice's client device 104, thereby preempting and preventing any such download of malware 130 to Alice's client device 104. As further described below, data appliance 102 performs the disclosed techniques for automatic detection of unknown / custom packers to detect such malware 130 and block it from harming Alice's client device 104.

[0051] In various embodiments, data appliance 102 is configured to operate in cooperation with security platform 122. As one example, security platform 122 can provide a set of signatures of known malicious files to data appliance 102 (e.g., as part of a subscription). If the signature of malware 130 is included in the set (e.g., the MD5 hash of malware 130), data appliance 102 can accordingly prevent the transmission of malware 130 to client device 104 (e.g., by detecting that the MD5 hash of an email attachment file sent to client device 104 matches the MD5 hash of malware 130). Security platform 122 also provides a list of known malicious domains and / or IP addresses to data appliance 102, enabling data appliance 102 to block traffic between enterprise network 140 and C&C server 150 (e.g., where C&C server 150 is known to be malicious). The list of malicious domains (and / or IP addresses) can also help data appliance 102 determine when one of its nodes has been exposed to danger. For example, if client device 104 attempts to contact C&C server 150, such an attempt is a strong indicator that client 104 has been exposed to malware (and remedial action, such as refraining from having client device 104 communicate with other nodes within enterprise network 140, should be taken accordingly).

[0052] As will be described in more detail below, the security platform 122 can also receive a copy of the malware 130 from the data appliance 102 and perform cloud-based security analysis to perform an automated detection of unknown / custom packers, and the malware determination is returned to the data appliance 102 to enforce the security policy, thereby protecting Alice's client device 104 from the execution of the malware 130 (e.g., blocking the malware 130 from access on the client device 104).

[0053] Furthermore, the security platform 122 can also provide the data appliance 102 with other types of information, such as a set of information for performing the techniques disclosed for the automated detection of unknown / custom packers available to the data appliance 102 for performing in-line analysis of such malware files (e.g., as part of a subscription), as will be further described below.

[0054] In various embodiments, when the signature of an attached file is not found, various actions can be taken by the data appliance 102. As a first example, the data appliance 102 can be fail-safe by blocking the transmission of any attached file that is not listed as benign in a whitelist (e.g., does not match the signature of a known good file). The drawback of this approach is that many legitimate attached files may be unnecessarily blocked as potential malware when they are actually benign. As a second example, the data appliance 102 can be fail-danger by allowing the transmission of any attached file that is not listed as malicious in a blacklist (e.g., does not match the signature of a known bad file). The drawback of this approach is that newly created malware (not previously seen by the platform 122) can cause harm and is not prevented. As a third example, the data appliance 102 can be configured to provide a file (e.g., malware 130) to the security platform 122 for static / dynamic analysis to determine whether it is malicious and / or, if not, to classify it.

[0055] The security platform 122 stores a copy of the received sample in the storage 142, and then analysis is started (or scheduled if applicable). One example of the storage 142 is an Apache Hadoop Cluster (HDFS). The results of the analysis (and additional information about the application) are stored in the database 146. If the application is determined to be malicious, the data appliance can be configured to automatically block file downloads based on the analysis results. Further, a signature for the malware is generated and distributed (for example, to the data appliances such as the data appliances 102, 136, and 148), and future file transfer requests for downloading files determined to be malicious can be automatically blocked.

[0056] In various embodiments, the security platform 122 comprises one or more dedicated, commercially available hardware servers (e.g., having a multi-core processor, 32G+ of RAM, a gigabit network interface adapter, and a hard drive) running a typical server-class operating system (e.g., Linux®). The security platform 122 can be implemented across a scalable infrastructure that includes multiple such servers, solid state drives, and / or other applicable high-performance hardware. The security platform 122 can comprise several distributed components, including components provided by one or more third parties. For example, part or all of the security platform 122 can be implemented using Amazon Elastic Compute Cloud (EC2) and / or Amazon Simple Storage Service (S3). Further, similar to the data appliance 102, whenever the security platform 122 is referred to as performing tasks such as storing or processing data, it should be understood that sub-components or multiple sub-components of the security platform 122 can cooperate (either individually or in cooperation with third-party components) to perform that task. As one example, the security platform 122 can optionally perform static / dynamic analysis in cooperation with one or more virtual machine (VM) servers, such as VM server 124.

[0057] One example of a virtual machine server is a commercial server-class hardware (e.g., a multi-core processor, 32+ gigabytes of RAM, and one or more Gigabit network interface adapters) that runs commercial virtualization software such as VMware ESXi, Citrix XenServer, or Microsoft Hyper-V, which is a physical machine. In some embodiments, the virtual machine server is omitted. Further, the virtual machine server may be under the control of the same entity that manages the security platform 122, or may be provided by a third party. As one example, the virtual machine server can rely on EC2, and the rest of the security platform 122 is owned by the operator of the security platform 122 and provided by dedicated hardware under the control of that operator. The VM server 124 is configured to provide one or more virtual machines 126-128 for emulating client devices. The virtual machines can run various operating systems and / or their versions. The observed behavior resulting from running an application within a virtual machine is logged (e.g., for indicators that the application is malicious) and analyzed. In some embodiments, the log analysis is performed by the VM server (e.g., VM server 124). In other embodiments, the analysis is at least partially performed by other components of the security platform 122, such as the coordinator 144.

[0058] In various embodiments, security platform 122 makes the analysis results of samples available to data appliance 102 as part of a subscription via a list of signatures (and / or other identifiers). For example, security platform 122 can periodically (e.g., daily, hourly, or at some other interval and / or based on one or more events configured by a policy) send content packages that identify malware files, including for the automatic detection of unknown / custom packers, etc. One exemplary content package includes a packer detector 154 and / or other information (e.g., a packer detection model) as further described below. The subscription can cover the analysis of only the files intercepted by data appliance 102 and sent by data appliance 102 to security platform 122, and can also cover malware signatures known to security platform 122. As described in more detail below, platform 122 can also make available other types of information for the automatic detection of unknown / custom packers using a packer detection model generated using model builder 152 and packer detector 154 (e.g., as related to FIGS. 4A and 4B as further described below). It is performed using the disclosed automatic custom / unknown packer detection techniques that can help data appliance 102 detect and perform in-line blocking of malware created using such custom / unknown packers.

[0059] In various embodiments, security platform 122 is configured to provide security services to various entities in addition to (or, where applicable, instead of) the operator of data appliance 102. For example, other enterprises having respective corporate networks 114 and 116, and respective data appliances 136 and 148, can contract with the operator of security platform 122. Other types of entities can also utilize the services of security platform 122. For example, an Internet service provider (ISP) providing Internet services to client device 110 can contract with security platform 122 to analyze applications that client device 110 attempts to download. As another example, the owner of client device 110 can install software that communicates with security platform 122 on client device 110 (e.g., receive a content package from security platform 122, use the received content package to check attached files according to the techniques described herein, and send the application to security platform 122 for analysis).

[0060] Analysis of Samples Using Static / Dynamic Analysis

[0061] Figure 3 shows one exemplary logical component that may be included in a system for analyzing samples. Analysis system 300 can be implemented using a single device. For example, the functionality of analysis system 300 can be implemented in malware analysis module 112 incorporated in data appliance 102. Analysis system 300 can also be implemented collectively across a plurality of separate devices. For example, the functionality of analysis system 300 can be provided by security platform 122.

[0062] In various embodiments, the analysis system 300 utilizes a list, database, or other collection of known safe content and / or known bad content (collectively shown as collection 314 in FIG. 3). The collection 314 can be obtained in various ways, including via a subscription service (e.g., provided by a third party) and / or as a result of other processes (e.g., executed by data appliance 102 and / or security platform 122). Examples of the information included in collection 314 are as follows. That is, URLs, domain names, and / or IP addresses of known malicious servers, URLs, domain names, and / or IP addresses of known safe servers, URLs, domain names, and / or IP addresses of known command and control (C&C) domains, signatures, hashes, and / or other identifiers of known malicious applications, signatures, hashes, and / or other identifiers of known safe applications, signatures, hashes, and / or other identifiers of known malicious files (e.g., OS exploit files), signatures, hashes, and / or other identifiers of known safe libraries, and signatures, hashes, and / or other identifiers of known malicious libraries.

[0063] In various embodiments, when a new sample is received for analysis (e.g., an existing signature associated with the sample does not exist in the analysis system 300), it is added to queue 302. As shown in FIG. 3, application 130 is received by system 300 and added to queue 302.

[0064] The coordinator 304 monitors the queue 302, and when a resource (e.g., a static analysis worker) becomes available, the coordinator 304 fetches a sample from the queue 302 for processing (e.g., fetches a copy of the malware 130). In particular, the coordinator first provides a sample to the static analysis engine 306 for static analysis (305). In some embodiments, one or more static analysis engines are included within the analysis system 300, where the analysis system 300 is a single device. In other embodiments, the static analysis is performed by a separate static analysis server that includes multiple workers (i.e., multiple instances of the static analysis engine 306).

[0065] The static analysis engine obtains general information about the sample and includes it in the static analysis report 308 (along with heuristic information and other information where applicable). The report can be created by the static analysis engine or by a coordinator 304 configured to receive information from the static analysis engine 306 (or by another suitable component). As one example, static analysis of malware can include performing signature-based analysis. In some embodiments, the information collected is stored in a database record for the sample (e.g., within database 316) instead of or in addition to the separate static analysis report 308 being created (i.e., portions of the database record form the report 308). In some embodiments, the static analysis engine also forms a verdict for the application (e.g., "safe", "suspicious", or "malicious"). As one example, if there is even one "malicious" static characteristic within the application (e.g., the application includes a hard link to a known malicious domain), the verdict can be "malicious". As another example, points can be assigned to each of the characteristics (e.g., based on severity if found, based on how reliable the characteristic is for predicting malice, etc.). And based on the number of points associated with the static analysis results, a verdict can be assigned by the static analysis engine 306 (or by the coordinator 304 where applicable).

[0066] Once the static analysis is complete, the coordinator 304 locates an available dynamic analysis engine 310 to perform a dynamic analysis on the application. Similar to the static analysis engine 306, the analysis system 300 can directly include one or more dynamic analysis engines. In other embodiments, the dynamic analysis is performed by a separate dynamic analysis server that includes a plurality of workers (i.e., multiple instances of the dynamic analysis engine 310).

[0067] Each dynamic analysis worker manages a virtual machine instance. In some embodiments, the results of the static analysis (e.g., performed by the static analysis engine 306) are provided as input to the dynamic analysis engine 310 regardless of whether they are in report form (308), stored in the database 316, or stored otherwise. For example, the static report information can be used to help select / customize the virtual machine instance (e.g., Microsoft Windows 7 SP 2 vs. Microsoft Windows 10 Enterprise, or iOS 11.0 vs. iOS 12.0) used by the dynamic analysis engine 310. If multiple virtual machine instances are being run simultaneously, a single dynamic analysis engine can manage all of the instances, or multiple dynamic analysis engines can be used, if applicable (e.g., each managing its own virtual machine instance). As will be described in more detail below, during the dynamic part of the analysis, actions (including network activities) performed by the application are analyzed.

[0068] In various embodiments, static analysis of the sample is either omitted or, if applicable, performed by a separate entity. As one example, conventional static and / or dynamic analysis can be performed on a file by a first entity. Once a given file is determined to be malicious (e.g., by the first entity), the file can be provided to a second entity (e.g., an operator of security platform 122) for additional analysis regarding the use of malware for network activity (e.g., by dynamic analysis engine 310).

[0069] The environment used by analysis system 300 is instrumented / hooked such that the behavior observed while the applications are running is logged (e.g., using a customized kernel that supports hooking and logcat) as they occur. Network traffic associated with the emulator is also captured (e.g., using pcap). The log / network data can be stored as temporary files on analysis system 300 and also more persistently (e.g., using HDFS or another suitable storage technology, or a combination of technologies such as MongoDB). The dynamic analysis engine (or another suitable component) can compare the connections made by the sample against a list of domains, IP addresses, etc. (314) and determine whether the sample communicated (or attempted to communicate) with a malicious entity.

[0070] Similar to the static analysis engine, the dynamic analysis engine stores the results of its analysis in a record associated with the application being tested in database 316 (and / or, where applicable, includes the results in report 312). In some embodiments, the dynamic analysis engine also forms a determination regarding the application (e.g., "safe", "suspicious", or "malicious"). As one example, even if one "malicious" action is taken by the application (e.g., an attempt is made to contact a known malicious domain, or an attempt to extract sensitive information is observed), the determination can still be "malicious". As another example, points can be assigned to the actions taken (e.g., based on the severity if found, based on how reliable the action is for predicting malice, etc.). And based on the number of points associated with the dynamic analysis results, a determination can be assigned by the dynamic analysis engine 310 (or, where applicable, the coordinator 304). In some embodiments, the final determination associated with the sample is made based on a combination of reports 308 and 312 (e.g., by the coordinator 304).

[0071] Automatic Detection of Unknown Packers

[0072] FIG. 4A shows a part of one exemplary embodiment of a packer detector for automatically detecting unknown packers according to some embodiments. As described above similarly, in various embodiments, the security platform 122 includes a packer detector 154. The packer detector includes a packer filter 410 that performs various operations to determine whether a given sample input 402 is a packed sample, as further described below. Samples determined to be packed samples are supplied to an emulator 420. The emulator performs various operations to extract predetermined features from the packed samples, as further described below. The extracted features of the packed samples are then provided to a cluster generator 430. The cluster generator uses the extracted features to perform a clustering operation and generate multiple clusters of the packed samples, as further described below. For example, packed samples having similar features are generally associated with a predetermined cluster, as further described below. Each of these components is implemented in Python and / or a high-level programming language and can use various open-source or publicly available components as further described below.

[0073] Referring to the packer filter 410, the packer filter includes an entropy calculation component 412. Generally, the packer filter reduces the number of samples for further analysis by identifying samples from a set of input samples (402) that are packed samples and / or highly likely to be packed samples (e.g., this packer filtering operation can reduce the processing volume for feature extraction and clustering operations by improving clustering quality and reducing false negatives, and thus perform further analysis only on the (likely) packed samples). The entropy calculator performs an entropy calculation based on the content of the binary code within the sample. If the result of the entropy calculation exceeds a threshold entropy (e.g., in one exemplary implementation, if the entropy is higher than 7, a determination that it is a packed sample can be made with confidence. For example, see https: / / www.semanticscholar.org / paper / Using-Entropy-Analysis-to-Find-Encrypted-and-Packed-Lyda-Hamrock / 86bf19a53e70d346d2105e4a3bcd9a588b943dcd / figure / 0), then the result is used as a factor to determine that the sample is likely a packed sample (e.g., packers generally increase the entropy associated with the content of the sample by using compression, obfuscation, and / or other techniques to pack the sample). There are various existing libraries in Python for determining entropy calculations for content such as files / samples (e.g., an exemplary Python library implementation for performing entropy calculations is publicly available at https: / / gist.github.com / jaradc / eeddf20932c0347928d0da5a09298147).

[0074] Referring to the packer filter 410, the packer filter also includes an import table check component 414. An abnormal import table component can be implemented as a set of heuristics for determining whether, for example, the import table associated with a Microsoft Windows PE file is abnormal. The determination of an abnormal import table is used as another factor for determining that a sample is likely to be a packed sample (for example, a packed sample generally is likely not to have the attributes associated with an unpacked Microsoft Windows PE file). As one example, an import table that imports a zero library can be considered abnormal (for example, suspicious and likely associated with packed malware).

[0075] In the implementation of heuristics / rules for one exemplary import table check component 414, the following heuristics can be used to identify suspicious Windows PE file / formats and / or abnormal import tables of Microsoft Windows PE files.

[0076] pe_sa_last_exec_sect: The last section of the PE has execute permission.

[0077] pe_sa_first_write_sect: The first section of the PE has write permission.

[0078] pe_sa_abnl_ep_addr: The entry point address is abnormal. For example, outside the file.

[0079] pe_sa_abnl_IAT: Abnormal import table format. This is one example of a heuristic / rule for detecting an abnormal import table.

[0080] pe_sa_abnl_sect_entropy: The entropy of the section is abnormal.

[0081] pe_sa_ol_entropy: Calculate the entropy of the overlay as an ML feature.

[0082] pe_sa_zero_size_sect: Count the number of zero-size sections as an ML feature.

[0083] pe_sa_mem_wx_sect: Count the number of sections with write / execute attributes as an ML feature.

[0084] pe_sa_err_delay_imp_dir: Whether there is an error during parsing the delay import directory as an ML feature. This is another example of a heuristic / rule for detecting an abnormal import table.

[0085] pe_sa_err_imp_dir: Whether there is an error during parsing the import directory. This is another example of a heuristic / rule for detecting an abnormal import table.

[0086] pe_sa_err_parsing_imp_table: Whether there is an error during parsing the delay import table. This is another example of a heuristic / rule for detecting an abnormal import table.

[0087] Referring to the packer filter 410, the packer filter also includes an overlay check component 416. The overlay check component determines whether a sample includes an overlay and whether that overlay is suspicious / potentially associated with a malware packer. An overlay generally refers to the last data of a Windows PE file and is not defined by the file header. Three exemplary features for checking an overlay are as follows: (1) having an overlay, (2) overlay size (e.g., to determine whether the overlay size is abnormally large or small), and (3) overlay entropy (e.g., an overlay entropy of less than 4 is plain text, an overlay entropy between 4 and 5 is executable, an overlay entropy between 6 and 7 is packed code, and an overlay entropy greater than 7 is encrypted data).

[0088] Referring to the emulator 420, the emulator component provides an instrumentation environment for emulating the first N instructions (e.g., the first 100 instructions, 500 instructions, 1000 instructions, etc.) at the entry point of a given sample containing executable code (e.g., a Microsoft Windows PE file, or other executable file formats can be similarly emulated). In an exemplary experiment (e.g., as described below with respect to FIGS. 5A and 5B), the emulator component was configured to emulate the first 100 instructions at the entry point of a Microsoft Windows PE sample file. The emulation environment is instrumented to monitor activities associated with the sample during emulation to extract various types of features that can later cluster packed samples, as described below.

[0089] Referring to emulator 420, the emulator includes an opcode feature component 422. The opcode feature component generates a feature vector for a given opcode. In one exemplary implementation of an opcode feature component for parsing Microsoft Windows PE files, a set of 112 distinct opcodes are selected as features, and the remaining opcodes are grouped into a single feature (e.g., out of approximately 1500 opcodes in the x86 computing platform, a subset of 112 opcodes were selected based on emulated analysis of 9 common malware packer families, where the 112 opcodes were the ones most frequently observed during the emulation of these 9 common malware packer families, and the remaining opcodes were simply counted as the catch-all "other opcodes" feature value in this opcode feature vector). Thus, the opcode feature vector includes a count for each of these given opcodes observed during the above-described emulation analysis (e.g., no-operation (NOP) instructions can be removed from this emulation analysis). These opcode features can then be used during the clustering operation (430).

[0090] An exemplary list of opcode features for the x86 computing platform is provided below (e.g., opcodes observed as being executed during the emulation analysis). instr_list = ['adc', 'add', 'and', 'bnd ret', 'bsf', 'bsr', 'bswap', 'bt', 'btr', 'bts', 'call', 'cbw' 'clc' 'cld' 'cmc' 'cmova' 'cmp' 'cpuid' 'cwde' 'dec' 'div' 'enter' 'fadd' 'fild' 'fistp' 'fld' 'fldcw' 'fnclex' 'fninit' 'fnop' 'fnstcw' 'fstp' 'fsub' 'imul' 'inc' 'int3' 'ja' 'jae' 'jb' 'jbe' 'je' 'jg' 'jge' 'jl' 'jle' 'jmp' 'jne' 'jno' 'jnp' 'jns' 'jo' 'jp' 'jrcxz' 'js' 'lar' 'lea' 'leave' 'lock add' 'lock cmpxchg' 'lodsb' 'lodsd' 'loop' 'mov' 'movapd' 'movsb' 'movsd' 'movsx' 'movsxd' 'movzx' 'neg' 'nop' #<-- this should be removed 'not' 'or' 'pop' 'popfq' 'push' 'pushf' 'pushfq' 'rcl' 'rcr' 'rep movsb' 'rep movsd' 'rep stosd' 'ret' 'rol' 'ror' 'sal' 'sar' 'sbb' 'scasb' 'seta' 'setbe' 'sete' 'setge' 'setne' 'shl' 'shld' 'shr' 'shrd' 'sidt' 'sldt' 'stc' 'stosb' 'stosd' 'str' 'sub' 'test' 'verr' 'wait' 'xchg', 'xor'

[0091] Referring to emulator 420, the emulator also includes a PE format feature component 424. The PE format feature component extracts various predetermined features associated with the Microsoft Windows PE file determined during the emulator analysis of the sample. These PE format features can then be used during the clustering operation (430). Exemplary PE format features include features related to the PE format structure, entropy, entry point, and size of the format (e.g., pe_sa_last_exec_sect, pe_sa_first_write_sect, pe_sa_abnl_ep_addr, pe_sa_abnl_sect_entropy, pe_sa_zero_size_sect, and pe_sa_mem_wx_sect).

[0092] Referring to emulator 420, the emulator also includes an API feature component 426. The API feature component extracts various predefined APIs called during the emulator analysis of the sample (e.g., API calls generally associated with virtual memory attacks, etc.). These API features can then be used during the clustering operation (430). In one exemplary implementation, the emulator can be implemented using an openly available binary emulation framework such as the one available at https: / / github.com / qilingframework / qiling.

[0093] A list of exemplary APIs (e.g., API features) for the x86 computing platform is provided below (e.g., APIs observed as being called during the emulation analysis). 'CharNextA' 'CharNextW' 'CreateDirectoryA' 'CreateFileA' 'CreateWindowExA' 'DecodePointer' 'EncodePointer' 'EnterCriticalSection' 'ExitProcess' 'FlsAlloc' 'FlsSetValue' 'GetCPInfo' 'GetCommandLineA' 'GetCommandLineW' 'GetCurrentProcessId' 'GetCurrentThreadId' 'GetEnvironmentVariableA' 'GetKeyboardType' 'GetLastError' 'GetLocaleInfoA' 'GetMessageA' 'GetModuleFileNameA' 'GetModuleHandleA' 'GetModuleHandleW' 'GetProcAddress' 'GetProcessHeap' 'GetStartupInfoA' 'GetStartupInfoW' 'GetSystemDirectoryA' 'GetSystemDirectoryW' 'GetSystemTimeAsFileTime' 'GetTempPathA' 'GetThreadLocale' 'GetTickCount' 'GetVersion' 'GetVersionExA' 'GetVersionExW' 'GetWindowsDirectoryA' 'HeapAlloc' 'HeapCreate' 'HeapFree' 'HeapSetInformation' 'HeapSize' 'InitCommonControls' 'InitCommonControlsEx' 'InitializeCriticalSection' 'InitializeCriticalSectionAndSpinCount' 'InitializeCriticalSectionEx' 'InterlockedCompareExchange' 'InterlockedExchange' 'InterlockedIncrement' 'IsBadReadPtr' 'IsDBCSLeadByte' 'IsProcessorFeaturePresent' 'LeaveCriticalSection' 'LoadCursorA' 'LoadIconA' 'LoadLibraryA' 'LoadLibraryExA' 'LoadLibraryExW' 'LoadLibraryW' 'LocalAlloc' 'MessageBoxA' 'OleInitialize' 'QueryPerformanceCounter' 'RegOpenKeyExA' 'RegisterClassExA' 'SHGetFileInfoA' 'SHGetFileInfoW' 'SetDefaultDllDirectories' 'SetDllDirectoryW' 'SetErrorMode' 'SetThreadLocale' 'SetUnhandledExceptionFilter' 'ShowWindow' 'ThunRTMain' 'TlsAlloc' 'TlsGetValue' 'TlsSetValue' 'UpdateWindow' 'VerSetConditionMask' 'VerifyVersionInfoA' 'VirtualAlloc' 'VirtualProtect' 'WSAStartup' '_EH_prolog' '_getmainargs' '_p_commode' '_p_fmode' '_set_app_type' '_controlfp' '_initterm' '_ismbblead' '_lock' '_lopen' 'lstrcpyA' 'lstrcpynA' 'lstrcpynW' 'malloc' 'wsprintfA'

[0094] Referring to the cluster generator 430, the cluster generator component receives the features extracted from the emulator 420 during the emulation analysis. The cluster generator uses the features to cluster the packed samples into families as shown in Figure 4A. In one exemplary implementation, density-based special clustering (DBScan) of applications with noise can be used to cluster the packed samples based on the feature distance perspective (for example, a publicly / commercially available implementation of DBScan as available at https: / / scikit-learn.org / stable / can be used. Also refer to https: / / scikit-learn.org / stable / modules / generated / sklearn.cluster.DBScan.html, and / or use another clustering algorithm in the same way to cluster the packed samples based on the extracted features such as k-means clustering, mean shift clustering, HDBSCAN clustering, etc.).

[0095] In experiments based on a sample input set of approximately 10,000 PE files, it was determined that approximately 5,900 of these samples were likely packed using a packer filter. Clustering of the 5,900 packed (likely) samples obtained based on the extracted features was performed using DBScan, and 53 different packer clusters were generated. It should be noted that the results also included previously unknown / custom packers that were not previously identified by security vendors (for example, not included in VirusTotal or other vendor data for known packers). Also, in-depth analysis of these packer clusters revealed interesting common unique features associated with the related packers analyzed during this experiment.

[0096] Accordingly, the generated clustering operation output, which provides the resulting packer clusters, can then be further analyzed to automatically extract unique signatures in order to efficiently and effectively identify each packer family (e.g., such packer families typically being interesting and having unique characteristics). These signatures can then be used to provide malware detection using the data appliance (102) and / or cloud-based security service (122).

[0097] Furthermore, the above-described techniques for automatically detecting unknown packers can be similarly implemented to provide a new in-line malware packer detection solution as described below with respect to FIG. 4B.

[0098] FIG. 4B shows a portion of one exemplary embodiment of an in-line packer detector for automatically detecting unknown packers according to some embodiments. In one exemplary implementation, the in-line packer detector 450 is implemented as a component of the malware analysis module 112 and is executed on the data appliance 102 (as shown, for example, in FIG. 1).

[0099] Referring to FIG. 4B, a new sample input 452 is received by the in-line packer detector. The new sample is analyzed using a packer filter 410, which can be implemented in the same manner as described above with respect to FIG. 4A. If the sample is determined to be (likely) packed, the sample is further analyzed using a known packer cluster checker 460 (e.g., if the sample is not determined to be (likely) packed, no further analysis is performed using the in-line packer detector).

[0100] Referring to the known packer cluster checker 460, the known packer cluster checker can receive periodic clustering data results from the security platform 122 (e.g., using the packer detector 154 as similarly described above with respect to FIGS. 1 and 4A).

[0101] If the known packer cluster checker determines that the sample matches an existing known packer cluster, then, as shown at 470, the resulting output is sent to a cloud security solution (e.g., the security platform 122 shown in FIG. 1). For example, the cloud security solution can utilize the resulting output to provide further analysis and can further update the known packer clustering data set, which can then be periodically distributed to an in-line packer detection solution as implemented in the data appliance 102 (e.g., an in-line packer detector component and a data appliance with a subscription to receive periodically updated known packer clustering information from the cloud security solution).

[0102] If the known packer cluster checker determines that the sample does not match an existing known packer cluster, then the sample is provided to the emulator 430, which can be implemented as similarly described above with respect to FIG. 4A. The features extracted during the emulation analysis are stored for clustering as shown at 480. The stored features can be used for further clustering (e.g., locally executed using the data appliance 102 and / or provided to a cloud security solution executed using the security platform 122 as similarly described above with respect to FIGS. 1 and 4A). The updated clustering results can then be utilized to determine whether the sample is associated with a known packer cluster.

[0103] The inline packer detector can generate a malware packer determination (450) based on the results of samples processed using the inline packer detector. The data appliance (102) can then execute an action based on a policy (e.g., a security policy) in response to the malware packer determination. For example, the data appliance can be configured to block access to and / or storage of a sample if the sample is determined to be associated with a known malware packer (e.g., to prevent an endpoint device from receiving, storing, opening, and / or executing the sample). Other exemplary actions can include logging a sample determined to be associated with a known malware packer, blocking / dropping the sample, warning the endpoint user and / or network / security administrator that the sample has been determined to be associated with a known malware packer, isolating the endpoint device associated with the sample, identifying the source IP address, URL, etc. associated with the sample as malicious (or potentially malicious), and / or various other actions can also be executed based on the policy.

[0104] Accordingly, the disclosed techniques for automatically detecting unknown packers can be implemented to use inline malware detection on the data appliance 102, etc. (e.g., and / or using a security agent executed on a protected endpoint, on endpoints such as client devices 104, 106, and 108). And, also, as described herein, it can also be executed as a cloud-based security service, such as using the security platform 122.

[0105] Various use cases for automatically detecting unknown packers are described below.

[0106] Use Case for Automatically Detecting Unknown Packers

[0107] Figures 5A and 5B show exemplary packer clusters determined based on experimental results for automatically detecting unknown packers according to some embodiments.

[0108] Referring to Figure 5A, a mysterious cryptor packer cluster was discovered based on experimental results for automatically detecting unknown packers. As shown, this malware packer performs various malicious operations as shown at 502, exhibits an unpacking flow as shown at 504 (e.g., first revealing a similar unpack flow), and exhibits various behaviors as aggregated at 506 (e.g., using similar junk instruction tricks such as NOP instructions, and meaningless calls, and jmp operations). This malware packer had not been previously identified by existing security solutions for detecting malware (e.g., VirusTotal had not previously identified this malware packer, and there was no existing packer signature for this previously unknown malware packer).

[0109] Referring to FIG. 5B, based on the experimental results for automatically detecting unknown packers, a Themida variant packer class was discovered. As shown, this malware packer performs various malicious operations as indicated by 552, and as shown by 554, first uses the same anti-debugging mechanism (e.g., initially includes the same call graph and uses the Int3 operation to trap the debugger), and then exhibits various behaviors as aggregated by 556. This malware packer has not been previously identified by existing security solutions for detecting malware (e.g., VirusTotal has not previously identified this malware packer, and there was no existing packer signature for this previously unknown malware packer).

[0110] Additional exemplary processes will now be described for the disclosed techniques for automatically detecting unknown packers.

[0111] Exemplary Process for Automatically Detecting Unknown Packers

[0112] FIG. 6 is a flowchart of a process for automatically detecting an unknown packer according to some embodiments. In some embodiments, process 600 shown in FIG. 6 is executed by a security platform and techniques similar to those described above, including the embodiments described above with respect to FIGS. 1-4B. In one embodiment, process 600 is executed by data appliance 102 described above with respect to FIG. 1, security platform 122 described above with respect to FIG. 1 (e.g., as a cloud-based security service), a virtual appliance (e.g., Palo Alto Networks' VM series virtualized next-generation firewall, CN series container next-generation firewall, and / or other commercially available virtual-based or container-based firewalls that may be similarly implemented and configured to execute the disclosed techniques), an SDN security solution, a cloud security service, and / or a combination of the foregoing described herein, or a hybrid implementation.

[0113] At 602, a sample is received for malware packer detection analysis. For example, data appliance 102 and / or security appliance 122 can receive one or more input samples for malware packer detection analysis, similar to that described above with respect to FIGS. 1-4B.

[0114] At 604, a packer filter is executed to determine whether each of the samples is packed. For example, the packer filter can be executed using the entropy calculation component 412, import table check component 414, and / or overlay check component 416 of packer filter 410, similar to that described above with respect to FIGS. 4A and 4B.

[0115] At 606, an emulator is executed to emulate each of the packed samples. For example, the emulator can be executed to extract various features for clustering based on the extracted features, similar to that described above with respect to FIGS. 4A and 4B.

[0116] At 608, the packed samples are clustered based on the extracted features. For example, a cluster generator can be used to cluster the packed samples based on the extracted features, similar to that described above with respect to FIGS. 4A and 4B. Also, similar to that described above, the newly identified malware packer (e.g., and / or malware packer family) can then be further analyzed, for example, to generate a new signature for automatically detecting such newly identified malware packer (e.g., and / or malware packer family).

[0117] FIG. 7 is a flowchart of an in-line process for automatically detecting an unknown packer according to some embodiments. In some embodiments, the process 700 shown in FIG. 7 includes the embodiments described above with respect to FIGS. 1-4B and is similarly executed by the security platform and techniques described above. In one embodiment, the process 700 includes the data appliance 102 described above with respect to FIG. 1, the security platform 122 described above with respect to FIG. 1 (e.g., as a cloud-based security service), virtual appliances (e.g., Palo Alto Networks' VM series virtualized next-generation firewall, CN series container next-generation firewall, and / or other commercially available virtual-based or container-based firewalls similarly implemented and configured to execute the disclosed techniques), SDN security solutions, cloud security services, and / or combinations of the foregoing described herein, or by a hybrid implementation.

[0118] At 702, samples for in-line malware packer detection analysis are received. For example, data appliance 102 (e.g., and / or security appliance 122, such as those for which in-line malware packer detection analysis can be similarly performed by a cloud security service) can receive one or more samples for in-line malware packer detection analysis, as similarly described above with respect to FIGS. 1-4B.

[0119] At 704, a packer filter is executed to determine whether each of the samples is packed. For example, the packer filter can be executed using the entropy calculation component 412, the import table check component 414, and / or the overlay check component 416 of the packer filter 410, as similarly described above with respect to FIGS. 4A and 4B.

[0120] At 706, the packed samples are compared to known malware packer clusters to determine whether the samples are associated with a known malware packer. For example, the known malware packer cluster comparison can be executed using the known packer cluster checker 460, as similarly described above with respect to FIG. 4B.

[0121] At 708, a response action based on malware packer determination is executed. Similar to what was described above with respect to FIG. 4B, a malware packer can be detected based on the determination / result of in-line packer detection analysis. In one exemplary implementation, the in-line packer detector can generate a malware packer determination based on the results of samples processed using an in-line packer detector (e.g., the in-line packer detector 450 as shown in FIG. 4B). The data appliance (102) can then execute an action based on a policy (e.g., a malware packer policy that can be stored in the policy 252 as shown in FIG. 2B) in response to the malware packer determination. For example, the data appliance can be configured to block access to and / or storage of a sample if the sample is determined to be associated with a known malware packer (e.g., to prevent an endpoint device from receiving, storing, opening, and / or executing the sample). Other exemplary actions can include logging a sample determined to be associated with a known malware packer, blocking / dropping the sample, warning the endpoint user and / or network / security administrator that the sample is determined to be associated with a known malware packer, isolating the endpoint device associated with the sample, identifying the source IP address, URL, etc. associated with the sample as malicious (or potentially malicious), and / or various other actions can also be executed based on the policy.

[0122] The foregoing embodiments have been described in some detail for purposes of clarity of understanding, but the present invention is not limited to the details provided. There are many alternative ways to implement the present invention. The disclosed embodiments are exemplary and not limiting.

Claims

1. A system comprising a processor and a memory, wherein the processor is configured to receive a plurality of samples for malware packer detection analysis, execute a packer filter to determine whether each of the plurality of samples is packed, emulate each of the packed samples to extract a plurality of features, and cluster the packed samples based on the extracted features, and the memory is coupled to the processor and configured to provide instructions to the processor, a system.

2. One or more of the plurality of samples include Microsoft Windows PE files, the system according to claim 1.

3. The plurality of features include opcode features, the system according to claim 1.

4. The plurality of features include file format features, the system according to claim 1.

5. The plurality of features include application programming interface (API) features, the system according to claim 1.

6. The plurality of features include at least two of opcode features, file format features, and application programming interface (API) features, the system according to claim 1.

7. The plurality of features include opcode features, file format features, and application programming interface (API) features, the system according to claim 1.

8. Detecting previously unknown malware packers is performed using a security platform of a cloud service, the system according to claim 1.

9. A method comprising: receiving a plurality of samples for malware packer detection analysis; executing a packer filter to determine that each of the plurality of samples is packed; emulating each of the packed samples to extract a plurality of features; and clustering the packed samples based on the extracted features. A method.

10. One or more of the plurality of samples include Microsoft Windows PE files, the method according to claim 9.

11. The plurality of features includes opcode features, The method according to claim 9.

12. The plurality of features includes file format features, The method according to claim 9.

13. The plurality of features includes application programming interface (API) features, The method according to claim 9.

14. The plurality of features includes at least two of opcode features, file format features, and application programming interface (API) features, The method according to claim 9.

15. The plurality of features includes opcode features, file format features, and application programming interface (API) features, The method according to claim 9.

16. Detecting a previously unknown malware packer is performed using a security platform of a cloud service, The method according to claim 9.

17. A computer program stored on a non-transitory computer-readable medium, including a plurality of computer instructions, When the plurality of computer instructions are executed, cause the computer to Receive a plurality of samples for malware packer detection analysis, Execute a packer filter, where each of the plurality of samples is packed, Emulate each of the packed samples and extract a plurality of features, Cluster the packed samples based on the extracted features, To be performed, Computer program.

18. One or more of the plurality of samples includes a Microsoft Windows PE file, The computer program according to claim 17.

19. The plurality of features includes at least one or more of opcode features, file format features, and application programming interface (API) features, The computer program according to claim 17.

20. Detecting a previously unknown malware packer is performed using a security platform of a cloud service, The computer program according to claim 17.

21. A system including a processor and a memory, The processor is Receive a sample for in-line malware packer detection and analysis, Execute a packer filter to determine whether the sample is packed, Compare with known packer clusters to determine whether the sample is associated with the known malware packer, and Execute a response action based on the determination of the malware packer, configured as follows, The memory is coupled to the processor and configured to provide instructions to the processor, system.

Images