Iterative Memory Analysis for Malware Detection

Abstract

To provide an iterative anti-malware scan method, a system, and a storage medium.SOLUTION: A method of iterative anti-malware analysis is performed upon running an executable file on a target computing system, and includes iterative techniques that combine static and dynamic analysis. Static analysis is performed with a tree of file attributes by which files are classified as malicious with specified accuracy. Dynamic analysis is performed iteratively if static analysis does not allow classifying the file with the specified accuracy. Dynamic analysis includes intercepting an operation of decrypting data blocks during execution of a file code and updating the tree of file attributes taking into account encrypted data blocks. Then, static analysis of the file is performed on the basis of the updated tree of file attributes. The steps of intercepting and updating the file are repeated until the file is classified with the specified accuracy.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to computer security, including static and dynamic analysis of malware and malware detection. [Background technology]

[0002] In computer security, malware detection strategies can be classified according to the stage of malware execution they intervene in: pre-execution, runtime, and post-execution. While useful in many cases, pre-execution ("PE") analysis has been difficult to perform against the wide collection of encrypted and / or packed malware samples.

[0003] One of the problems is the lack of solutions for performing PE analysis for such encrypted or packed samples. Common methods use behavioral runtime methods or event collection. In such cases, PE analysis is underutilized and detection rates are not improved.

[0004] Here, "static analysis" refers to a program analysis technique based on software structure. Traditional static analysis methods parse and disassemble portable executable files to create attribute trees that include certificates, headers, libraries, imports, names, sizes, and other attributes. These trees are used to classify executable files as malicious or not. However, when malware is encrypted, typical static file analyzers face challenges. The encryption prevents static analysis from effectively analyzing the code.

[0005] When static analysis is ineffective, dynamic analysis may be used instead. The goal of dynamic analysis is to test untrusted files by running them in a safe environment to reveal the true nature of the file as it executes. Examples of traditional dynamic behavioral analysis tools include emulation, sandboxing, and similar methods.

[0006] Although emulators can deeply analyze applications, they have some limitations. Emulators cannot emulate the original system with 100% accuracy. Some malware can test the runtime environment and stop execution if an emulator is detected. Furthermore, emulators use system resources and require a significant amount of time for analysis. This is well known, and some malicious files contain dummy loops to extend execution and emulation times. These loops cause the emulator to finish its analysis before the malicious code is executed.

[0007] A sandbox is an isolated, secure environment used for file execution testing and analysis. Like emulators, sandboxes do not exactly match the target computer system. As a result, potentially malicious files may run differently in a sandbox than they would in the target environment. Sandboxing is also a resource-intensive process that cannot be performed efficiently in a real-time environment. Nevertheless, sandboxes are sometimes used in addition to more lightweight static analysis methods.

[0008] A system behavior analyzer is a method of controlling the execution of files on a target computer. It tracks API calls, threads, read and write operations, and all other operations that help detect malware. However, the problem is that such methods are not lightweight and put a strain on the system, slowing down performance. Summary of the Invention

[0009] To overcome the problems found in the prior art, static and dynamic analysis can be combined to analyze potentially malicious files, including encrypted files. For example, in static analysis, the entropy of the file body can be estimated to reach a tentative conclusion that the file is suspicious because many data blocks are encrypted. After reaching a tentative conclusion, dynamic methods are used to refine the conclusion. Dynamic methods reveal the nature of untrusted files by revealing their execution behavior. For example, if a file overwrites its own header when it begins execution or has other attributes that hide its nature, this observed behavior contributes to the conclusion that the file is malicious. Classification can be based on machine learning techniques.

[0010] Various system-level interceptors are used to re-estimate verdicts for the application at special control flow points such as thread / process creation, file mapping, etc. The verdicts re-estimated by the trained machine learning PE analysis model use data extracted from the application memory rather than from files on disk. In these situations, the application memory contains new code regions where the decrypted code executes. Several iterations are performed to achieve more accurate results. There are also methods that leverage the synergy between static and dynamic analysis to achieve more accurate verdicts than traditional methods.

[0011] The method employs a PE analysis service, a driver, a file system, and a request to the file system. The driver communicates with the user-mode service to invoke the PE model evaluation and obtain the latest PE analysis verdict. The driver can utilize user-mode hooks to define control points in the code flow and re-estimate the model.

[0012] New code regions containing unpacked or decrypted code are detected using stack analysis and user-mode hooks that operate on the process virtual memory API, and are added to the model's dataset to train the model or compute new decisions.

[0013] Alternatively, the new code region is collected in the Portable Executable's uninitialized data segment, also known as the "BSS" segment, which acts as a placeholder for uninitialized variables that do not require actual disk space.

[0014] A similar combination of drivers and user-mode hooks is used to train a PE analysis model using a set of packed or encrypted malware samples. The updated PE analysis verdicts are then used by endpoint security products to improve malware detection.

[0015] In one embodiment of the present invention, an anti-malware scan of a file is performed upon execution of an executable file on a target computer system. Static analysis of the file is performed using a tree of file attributes that allows the file to be classified as malicious with a specific degree of accuracy. If the static analysis fails to classify the file with the specific degree of accuracy, dynamic file analysis is iteratively performed. The dynamic analysis involves intercepting operations to decrypt data blocks during execution of the file code and updating a tree of file attributes to take into account the encrypted data blocks. Static analysis of the file is then performed based on the updated tree of file attributes. The intercepting and updating steps are repeated until the file can be classified with the specific degree of accuracy.

[0016] In one embodiment, the data block is decoded into existing memory allocated to the application, into a new buffer in the allocated memory, or into an existing buffer of the application. A buffer is a temporary storage area, usually a block in memory, where items are placed while waiting to be transferred or modified from an input device or to an output device, or from another memory block for processing.

[0017] In one embodiment, the steps of intercepting the decoding of the data block and updating the file attribute tree are repeated more than three times, more than five times, or more than ten times.

[0018] In one embodiment, analysis of a file relies on a second, untrusted file being modified from the file after execution, resulting from execution of the first file changing its header, data blocks, or file body, or creating a new file.

[0019] In one embodiment, the execution of the file is frozen after analyzing the added file data. In one embodiment, a predetermined region of the buffer is made executable and execution is switched to the predetermined region. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 shows an overview of a system for implementing PE analysis. [Figure 2] FIG. 1 illustrates a process for performing an iterative static-dynamic analysis on an unknown object. [Figure 3] FIG. 1 illustrates the process of performing iterative static-dynamic analysis on unknown objects in machine learning. [Figure 4]FIG. 2 illustrates details of one embodiment of an iterative static-dynamic analysis performed in an execution environment. DETAILED DESCRIPTION OF THE INVENTION

[0021] Iterative malware analysis implementations use static and dynamic techniques to overcome obstacles to detection that are embedded in the malware by malware authors who are familiar with the detection methods used by anti-malware systems.

[0022] Iterative malware analysis is designed to selectively use static and dynamic techniques to improve detection rates and thereby overcome anti-detection characteristics found in advanced malware. For example, this may include methods to evade detection when the main malicious code in an untrusted file is executed. If malware can detect a virtualized runtime environment that does not match the target environment, it can evade dynamic methods such as emulation and sandboxing. Another evasion method used by malware is obfuscation. Obfuscation takes code and makes it unreadable by traditional means. It also delays detection and makes the code less susceptible to reverse engineering. Packing is a type of obfuscation used by malware that utilizes a packer. A packer is a tool that modifies the format of code by compressing or encrypting data.

[0023] Encrypted or packed files present a different challenge than traditional malware analysis protocols known from the prior art. When encrypted malware is executed, it first loads and executes a suspected malicious file. Typically, the body is encrypted or packed, making it impossible to determine the nature of the file with sufficient accuracy. To overcome this, data blocks are decrypted in a new buffer in allocated memory. Alternatively, data blocks can be decrypted in an existing buffer, making a predetermined region executable and switching execution to the predetermined region. This cycle can be repeated multiple times before the malicious code begins execution. Thus, decryption proceeds step by step, with data blocks being read from the encrypted data blocks and written to existing application memory or a new buffer.

[0024] Alternatively, the Uninitialized Data Segment (BSS) is used for decryption. Packing is often used for executable files. The packing process generates a new header and can transform the original file. The transformed original file is copied to a new file. A stub is added to the new file. If the file is executable, the stub reverses the transformation during execution.

[0025] Microsoft Windows® program files use the Portable Executable format. This format was first introduced in Windows NT 3.1 and has been expanded to other binary formats, including .NET and 64-bit support. The Portable Executable format is the standard binary format for EXE and DLL files in the Windows OS. The Linux® Executable and Linkable Format ("ELF") is a similar format for executable files in the Linux environment. In the following description, the Windows format is used as an example, but the techniques described below can be used with any structurally similar file format.

[0026] Process hollowing is a method of executing code within the address space of a live process. It is typically performed by creating a suspended process and then unmapping or "hollowing" the memory. This process hollowing can occur within the Uninitialized Data Segment (BSS). Detection can be achieved by setting the execute bit to allow access to the containing directory.

[0027] Portable executable header overwriting is a technique used by malware at runtime to evade detection. For example, the original file contains a header, a code section, and other sections such as .data or .rsrc. During the packing process, an entirely new executable file with a new header can be generated. The original file can be transformed by compression, encryption, or a combination of these. When loaded, the packed file begins and is mapped into virtual memory. A decompression stub is called to reverse the compression or encryption operation, if necessary. The original file is then restored in memory, resulting in a new header. Any changes to the file header at runtime are evidence that the file has been packed.

[0028] Static and dynamic analysis techniques have different impacts on overall system performance. For example, static PE analyzers can achieve high detection rates when decoding data from portable executable files (PE files) loaded into memory, while also classifying PE files more quickly and efficiently than dynamic analysis of the same files.

[0029] FIG. 1 is a diagram illustrating an overview of a system 100 for implementing PE analysis. The anti-malware system 100 includes a PE analyzer hosting service 102 for analyzing potentially malicious applications 104. The PE analyzer service 102 generates on-access scans 106 starting with iteration 0 and continuing through iterations 1 (108) through N (110). These elements, along with a user mode UM hook 112, are located in user mode 114.

[0030] Kernel mode 116 includes a kernel driver 122 that processes input / output I / O commands from UM hook unit 112 and a link 120 between results 122 of UM hook unit 112 and PE analysis service 102. Link 120 passes results 124 to PE analysis unit 102. Requests 126 from PE analysis service 102 include requests for process regions from PE analysis memory. User mode is distinguished from kernel mode in that applications run in user mode and core operating system components run in kernel mode.

[0031] FIG. 2 illustrates a method 200 for performing iterative static-dynamic analysis on an unknown file. The process begins by intercepting the start of file execution (step 202) and then obtaining file attributes (step 204). Based on the obtained attributes, static analysis is performed (step 206). The preliminary results of the static analysis produce a security ranking 208 and attribute-specific file classification 210 using machine learning algorithms on a collection of known files, allowing for matching 212 of file attributes with known malware in blacklists of suspicious certificates, libraries, code structures, and hash sums.

[0032] The static analysis decision 214 results from the ranking 208, classification 210, and matching 212. This decision may or may not be accurate enough to classify the target file according to a predetermined confidence level (or a specific accuracy). The confidence level may depend on the security policy and the class of the file. For example, a high level of security should correspond to a confidence level of 1% or less of a Type I error, while a medium level of security should correspond to a confidence level of 1% or less of a Type II error. If step 216 determines that the file can be classified as malware with a specific accuracy, execution of the file is terminated (step 218), and the detection of a malicious file is recorded (step 220). If decision 214 does not result in classification, a decision is made according to decision 214 as to whether further verification is required (step 222). If, on the other hand, classification is possible, file analysis is stopped, and the file is released for execution (step 224).

[0033] If the security ranking 208, file classification 210, or file attribute matching 212 with known malware in a blacklist of suspicious certificates, libraries, code structures, and hash sums cannot identify malware with a certain degree of accuracy, further verification is required. If further verification is required to classify the file, for example, if the confidence level is lower than a predefined level, or if other malware detection engines produce different classification verdicts, an iterative process is performed, beginning with step 226, with portions of the file running. File attributes are further identified using hooking operations (step 228). Hooking operations modify or extend a running application by intercepting function calls, messages, or events between software components. Code used for this purpose is commonly known as a hook.

[0034] The execution of the file is stopped in step 230 and the PE file attributes previously collected based on the static analysis generated in step 206 are updated (step 232). In iterative malware analysis, execution is stopped so that the execution of a single file can be tested up to N times without fully executing the entire code for the file in question.

[0035] FIG. 3 illustrates a method 300 for performing iterative static-dynamic analysis of unknown files in machine learning. The process begins with a new PE file analysis (step 302). PE file attributes are received (step 304) and evaluated (step 306). For example, if the attribute set allows for file classification with a higher confidence level than a predefined confidence level, or if the attribute set consists of a minimal, variable set of attributes, it is determined whether the attribute set is sufficient to train the system to identify files of interest. If not, the PE file is executed in a secure computing environment (step 308). Examples of secure computing environments include emulators, sandboxes, dedicated test machines, machines disconnected from any network, or other environments that limit the ability of malware to propagate. Hooking is performed to identify additional file attributes (step 310). Based on these additional attributes, the original set of attributes is updated (step 312). The iterative process continues until the evaluation in step 306 produces a positive result. If a positive result is obtained, the collected files (including the newly collected PE file-based attributes) are used to train a machine learning file analyzer (step 314). Based on the collected files and attributes, the detection rule set is updated (step 316).

[0036] FIG. 4 details one embodiment of an iterative static-dynamic analysis 400 performed in an execution environment 402. This environment includes system objects operating in user mode 404 and kernel mode 406: a PE analyzer 408 and an application 410 operating in user mode 404, and a kernel security driver 434. User mode is distinguished from kernel mode in that applications operate in user mode and core operating system components operate in kernel mode. The application 410 executes from a PE file 412 containing an encrypted data block 414. As the application 410 executes, it exposes code 416, libraries 418, and data 420. Block decryption is performed iteratively within buffer 0 (422), executing decryption block 426 and continuing to buffer N (428) and decrypted block 430. Alternatively, block decryption can occur within data block 420.

[0037] The hooking unit 432 runs in user mode 404 under the control of a kernel security driver 434. As with the other embodiments, the hooking unit 432 modifies or extends a running application by intercepting function calls, messages, or events between software components and sending them to the kernel security driver 434. The kernel security driver 434 sends the commands, messages, or events to the PE analyzer 408.

[0038] The initial PE file attributes 436 are collected by the PE analyzer 408 through a process of N iterations collecting PE file attributes 440, along with first iteration collected PE file attributes 438. The file attribute malware analyzer 442 evaluates the initial PE file attributes 436 and the iteratively collected file attributes 438, 440 based on instructions from the iteration controller 444.

[0039] Some malware does not create a buffer for block decryption. In such cases, data can be decrypted in existing memory pages. For example, code block 416 and data block 420 are used for decryption.

[0040] While some unpacked files restore the original file in memory, other malware restores the original file to a different memory address. In these cases, the activity associated with the second file is analyzed immediately after it begins, focusing on modifications from the first file.

[0041] Alternatively, another technique involves freezing further execution of the file after any additional data (such as data blocks, headers, code blocks, or data) written to the application buffer has been retrieved. For example, all threads of the process can be frozen.

[0042] The iterative collection of PE file attributes using the dynamic method continues until a tree of file attributes is created that is sufficient to draw conclusions about the file of interest. The repetition techniques described above are applicable to computing environments where user machines or networks are likely to encounter malware that has the ability to evade detection, particularly in ways that make static analysis unreliable or incomplete.

Claims

1. 1. A computer-implemented iterative anti-malware scanning method running on a processor, comprising: a. performing an anti-malware scan of the file when the file is executed on the target computer system; b. performing a static analysis of the file, the static analysis operating with a tree of file attributes that allows the file to be classified as malicious with a particular degree of precision; c. if the static analysis fails to classify the file to the specified accuracy, performing an iterative dynamic file analysis, d. intercepting the operation of decrypting the data block during execution of the file code; e. updating the file attribute tree to take into account the decoded data block; f. performing a static analysis of the file based on the updated file attribute tree; and g. repeatedly performing the dynamic file analysis, including repeating steps d, e, and f, until the file can be classified with the specified accuracy.

2. The method of claim 1 , further comprising repeating the intercepting and updating steps three or more times.

3. 3. The method of claim 2, further comprising the step of decoding the data block in the new or existing memory.

4. The method of claim 1 , further comprising analyzing the file and a second untrusted file that was modified after execution of the file.

5. 4. The method of claim 3, further comprising freezing execution of a second untrusted file after analyzing the file data added to the second untrusted file.

6. The method of claim 4 , further comprising repeating the steps of intercepting and updating at least three times.

7. 5. The method of claim 4, further comprising the step of decoding the data block in the new or existing memory.

8. 4. The method of claim 3, further comprising the steps of enabling a predetermined region and switching execution to said predetermined region.

9. A computer-readable storage medium having stored thereon program code, the program code, when executed by a computer, causing the computer to perform iterative static and hybrid virus scans, the iterative static and hybrid virus scans: a. performing an anti-malware scan of a file upon execution of the file on a target computer system; b. performing a static analysis of the file, the static analysis operating with a tree of file attributes that allows classifying the file as malicious with a certain degree of precision; c. if the static analysis fails to classify the file to the specified accuracy, iteratively performing dynamic file analysis, d. intercepting an operation to decrypt a data block during execution of the file code; e. Updating the file attribute tree to take into account the encrypted data block; f. performing a static analysis of the file based on the updated file attribute tree; and g. iteratively performing said dynamic file analysis, including repeating said intercepting step d, said updating step e, and said performing static analysis step f of said file until said file can be classified with said specified accuracy.

10. 10. The computer-readable storage medium of claim 9, further comprising repeating the intercepting and updating steps at least three times.

11. 11. The computer-readable storage medium of claim 10, further comprising the step of: decoding the data block in the new or existing memory.

12. 12. The computer-readable storage medium of claim 11, further comprising analyzing the file and a second untrusted file that was modified after execution of the file.

13. The computer-readable storage medium of claim 11 , further comprising the step of freezing execution of the file after analyzing the added file data.

14. 13. The computer-readable storage medium of claim 12, further comprising analyzing the file and a second untrusted file that was modified after execution of the file.

Images