A method for locating and extracting shellcode from the stack space under Windows x86
By hooking system API functions and using stack backtracking technology, shellcode in Windows x86 systems can be quickly located and extracted, solving the problem of low efficiency in manual extraction and making it suitable for batch virus sample detection.
Patent Information
- Application Number
- CN202310212104.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-07
- Publication Date
- 2026-02-24
- Estimated Expiration
- 2043-03-07
AI Technical Summary
In existing technologies, manually extracting shellcode from Windows x86 systems is inefficient, making it difficult to quickly locate and extract shellcode, especially when dealing with a large number of virus samples, which affects detection efficiency.
By hooking the system API functions LdrLoadDll, LdrGetProcedureAddressForCaller, NtCreateUserProcess, and ZwTerminateProcess, and combining stack backtracking and inline assembly, shellcode can be identified and extracted.
It enables fast and intuitive shellcode location and extraction. Although not completely accurate, it can cover the main body of the shellcode with a high probability, avoiding the inefficiency of manual extraction and is suitable for batch virus sample detection.
Smart Images

Figure CN116827585B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of network security technology, and in particular to a method for locating and extracting shellcode from the stack space under Windows x86. Background Technology
[0002] On the Windows x86 architecture, executing malicious code via stack overflow is a common attack method, especially on older versions of Windows systems with weaker security, posing a serious threat to the information security of both individuals and businesses.
[0003] For some viruses that exploit stack overflow to execute shellcode within the stack space, quickly obtaining their shellcode during dynamic analysis of their samples can provide a more concrete understanding of their behavioral characteristics.
[0004] Currently, there is a relative lack of information on shellcode extraction. Many resources only focus on dynamic or static detection of shellcode. For example, a shellcode detection method and apparatus published on the Chinese patent website (CN 104008336 B) uses document analysis and instruction simulation technology to parse the file to be detected, then uses simulated instructions to determine if there are suspicious instruction sequences, and compares them with a pre-established instruction sequence feature library to determine if shellcode is contained. However, most methods for locating and extracting shellcode rely on experienced technicians manually extracting it one by one. When faced with hundreds of files, the efficiency of manual extraction drops sharply, while the location and extraction of shellcode are crucial for detection. Summary of the Invention
[0005] To address the shortcomings of existing shellcode location and extraction methods, this invention proposes a method for locating and extracting shellcode from the stack space under Windows x86.
[0006] This invention proposes a method for locating and extracting shellcode in the stack space under Windows x86, including step one: hooking the system API functions LdrLoadDll, LdrGetProcedureAddressForCaller, NtCreateUserProcess, and ZwTerminateProcess.
[0007] Step 2: After a hit, perform stack backtracking on the hooked function. For each stack frame backtracked, check the location pointed to by the return address at that level to see if that location belongs to the stack space. If no return address pointing to the stack space is found after backtracking to the end, the function call is considered safe; otherwise, it is assumed that shellcode is being executed on the stack, and proceed to Step 3.
[0008] Step 3: When a certain level return address is found to point to the inside of the stack space, record this address value r, and record the EBP value of the nearest level along the address decreasing direction pointed to by r, and record it as e_1. Then, backtrack one level through e_1 to get e_2. At this time, e_2 is the theoretical EBP value of the current stack frame.
[0009] Step 4: Confirm the boundaries of the extraction.
[0010] Step 5: Based on the boundary range determined in Step 4, extract the binary data and then use a disassembler to identify it as x86 assembly, in which the actual shellcode will be contained.
[0011] Preferably, the stack backtracking method in step two is as follows:
[0012] First, the value of the EBP register is obtained through inline assembly in the hook function of the target function. The inline assembly looks like this:
[0013]
[0014] Here, dwEBP is a DWORD type variable used to receive the value of the EBP register. At this time, the value of EBP is used as a pointer to point to the bottom of the stack frame of the current function. The value stored at the bottom of the stack frame is the value of the EBP register when the previous function is called. This value is used as a pointer to point to the bottom of the stack frame of the previous function, and so on, so that the bottom of all stack frames of the current thread can be accessed.
[0015] The above technical solution determines the backtracking process based on the following criteria: the return address and EBP value of the nth stack frame are both 0, and the (n-1)th stack frame's return address points to the ntdll.dll module.
[0016] Preferably, the method for obtaining the function return address is as follows:
[0017] Adding 4 to the EBP value creates a pointer that points to the return address of the current function. Similarly, adding 4 to the EBP value of the parent function's stack frame creates a pointer that points to the return address of the parent function.
[0018] Preferably, the method for determining the stack space in step three is as follows:
[0019] The top and bottom addresses of the stack space are obtained by using the thread environment block of the current thread. The top of the stack is located at an offset of 4 bytes from the thread environment block, and the bottom of the stack is located at an offset of 8 bytes. The inline assembly code is as follows;
[0020]
[0021] Here, `buttom` and `top` are variables that accept the values at the bottom and top of the stack, respectively. The area pointed to by the values between the bottom and top of the stack is the stack space of the current thread.
[0022] Preferably, the method for determining that the return address points to the stack space is as follows:
[0023] During the backtracking process, the return address of each stack frame is checked. If the value of the return address is between the bottom and top of the stack, then the return address points to the stack space.
[0024] Preferably, the boundary in step four is divided into an upper boundary and a lower boundary, and the method for determining the upper boundary is as follows;
[0025] Starting from the address pointed to by r, the system scans byte by byte in the direction of decreasing memory size. When a preset condition is encountered during the upper bound scan, the scan stops, and the corresponding upper bound is determined.
[0026] Preferably, when a preset condition is encountered during the upper boundary scan, the scan is stopped, and the corresponding upper boundary is determined, including:
[0027] P1. At least 5 consecutive 0x00s, with the address of the first 0x00 plus 1 as the upper bound;
[0028] P2. At least 5 consecutive 0xcc addresses, with the address of the first 0xcc address plus 1 as the upper bound;
[0029] P3, at least 4 consecutive nop instructions (0x90), with the address of the first 0x90 plus 1 as the upper bound.
[0030] P4. The upper bound is the address where the r value is stored plus 4.
[0031] Preferably, the method for determining the lower bound is as follows:
[0032] Starting from the address pointed to by r, the system scans byte by byte in the direction of increasing memory size. When a preset condition is encountered during the lower bound scan, the scan stops, and the corresponding lower bound is determined.
[0033] Preferably, when a preset condition is encountered during the lower bound scan, the scan is stopped, and the corresponding lower bound is determined, including:
[0034] S1. At least 5 consecutive 0x00s, with the address of the first 0x00 minus 1 as the lower bound;
[0035] S2. At least 5 consecutive 0xcc addresses, with the address of the first 0xcc address minus 1 as the lower bound;
[0036] S3, at least 4 consecutive nop instructions (0x90), with the address of the first 0x90 minus 1 as the lower bound.
[0037] S4. If the function traces back from ZwTerminateProcess, then the address r is the ending address.
[0038] S5. Perform stack backtracking with e_2 as the initial value. If the backtracking is successful, the lower bound address is the address where e_2 is stored minus 4.
[0039] The beneficial effects of this invention are as follows:
[0040] This invention combines common behavioral characteristics of shellcode and hooks the system API functions that shellcode is most likely to call, achieving an intuitive and quick way to locate and extract shellcode. Although the extraction range cannot be completely accurate, it can include the main part of the shellcode with a high probability. This solution effectively avoids the technical problem of low efficiency in manual extraction. Attached Figure Description
[0041] Figure 1 This is a stack backtracking diagram illustrating the principle of a method for locating and extracting shellcode in the stack space under Windows x86 proposed in this invention.
[0042] Figure 2 This is a schematic diagram illustrating the principle of stack overflow execution of shellcode, a method for locating and extracting shellcode in the stack space under Windows x86 proposed in this invention.
[0043] Figure 3 This is a schematic diagram illustrating the determination of the upper bound of shellcode in the stack space under Windows x86 when shellcode performs an auxiliary attack through a large number of NOP instructions, as proposed in this invention.
[0044] Figure 4 This is a schematic diagram illustrating how the upper bound of shellcode is determined by using the return address of the stack frame as a reference in a method for locating and extracting shellcode in the stack space under Windows x86 proposed in this invention.
[0045] Figure 5This is a schematic diagram illustrating the method for determining the lower bound of shellcode when backtracking using the ZwTerminateProcess function in a method for locating and extracting shellcode in the stack space under Windows x86 proposed in this invention.
[0046] Figure 6 This is a schematic diagram illustrating the method for determining the lower bound of shellcode when the shellcode can automatically repair stack balance in a method for locating and extracting shellcode in the stack space under Windows x86 proposed in this invention. Detailed Implementation
[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0048] Reference Figure 1-6 A method for locating and extracting shellcode in the stack space under Windows x86 includes step one: hooking the system API functions LdrLoadDll, LdrGetProcedureAddressForCaller, NtCreateUserProcess, and ZwTerminateProcess.
[0049] LdrLoadDll is the lowest-level function for loading dynamic libraries. It is a function that is called when the shellcode program is executed, either alone or together with other functions.
[0050] LdrGetProcedureAddressForCaller is the lowest-level function that retrieves the address of an API function from a dynamic library. It is a function that is called when the shellcode program is executed, either alone or together with other functions.
[0051] NtCreateUserProcess is the lowest-level function for creating a process. It is a function that is called when the shellcode program is executed, either alone or together with other functions.
[0052] ZwTerminateProcess is the lowest-level function for terminating a process. It is a function that is called when the shellcode program is executed, either alone or together with other functions.
[0053] The four functions mentioned above are functions that the shellcode will inevitably call during execution, so it is necessary to hook these functions.
[0054] Hook technology, also known as hook function, involves a hook program capturing a message before the system calls the function. The hook function gains control and can then process (modify) the function's execution behavior or forcibly terminate message delivery. Simply put, it extracts the system's code and turns it into our own executable code snippets.
[0055] These four API functions are sub-functions of functions that almost all shellcode must access to execute successfully, because these four functions are low-level API functions of the Windows system, and all programs running on the Windows system may directly or indirectly call them.
[0056] Step 2: After a hit, perform stack backtracking on the hooked function. For each stack frame backtracked, check the location pointed to by the return address at that level to see if that location belongs to the stack space. If no return address pointing to the stack space is found after backtracking to the end, the function call is considered safe; otherwise, it is assumed that shellcode is being executed on the stack, and proceed to Step 3.
[0057] It should be noted that the security assessment in step two is not an absolute security assessment. It can only confirm that the attack methods of concern in this solution have not occurred. If they do occur, other attack methods may cause it to be directed to other illegal locations, but this situation is not the focus of this paper and therefore will not be considered.
[0058] The criteria for determining when to backtrack to the beginning are: the return address and EBP value of the nth stack frame are both 0, and the return address of the (n-1)th stack frame points to the ntdll.dll module.
[0059] like Figure 1 As shown, the stack backtracking method is as follows: First, in the hook function of the target function, the value of the EBP register is obtained through inline assembly. The inline assembly form is as follows:
[0060]
[0061] The above series of operations copies the value of the ebp register to our own variable dwEBP.
[0062] Here, `dwEBP` is a DWORD variable used to hold the value of the EBP register. Using the EBP value as a pointer, it points to the bottom of the current function's stack frame. The bottom of the stack frame stores the value of the EBP register at the time of the previous function call. Using that value as a pointer again points to the bottom of the previous function's stack frame, and so on, allowing access to the bottom of all stack frames of the current thread. The reason for not directly copying from the EBP register to the `dwEBP` variable is that this does not conform to the syntax rules and would not compile.
[0063] Additionally, to obtain the function return address, the specific method is as follows: add 4 to the value of EBP to create a pointer, which points to the return address of the current function; add 4 to the value of EBP of the previous function's stack frame to create a pointer, which points to the return address of the previous function, and so on. For example... Figure 1 As shown.
[0064] Step 3, as follows Figure 1 As shown, when a certain level return address is found to point to the inside of the stack space, this address value r is recorded, and the EBP value of the nearest level along the address decreasing direction is also recorded. This level EBP value is stored by the variable of the most recent backtracking, and is denoted as e_1. Then, backtracking one level through e_1 yields e_2, at which point e_2 is the theoretical EBP value of the current stack frame.
[0065] The above e_1 refers to the current EBP value, that is, the last EBP value obtained during the stack backtracking process at the current moment, the latest EBP value obtained.
[0066] e_2 refers to the EBP value obtained by backtracking the stack space again after obtaining e_1, which is more recent than e_1. That is, the value of e_1 is the address where the value of e_2 is stored.
[0067] The address where the value of e_1 is stored is greater than r, and the address where the value of e_2 is stored is less than r.
[0068] The specific method for determining the stack space is as follows: obtain the top and bottom addresses of the stack space through the current thread's thread environment block. The top of the stack is located at an offset of 4 bytes from the thread environment block, and the bottom of the stack is located at an offset of 8 bytes. The inline assembly code is as follows:
[0069]
[0070]
[0071] Here, `buttom` and `top` are variables that receive the values at the bottom and top of the stack, respectively. The regions pointed to by the values between these two are the stack space of the current thread.
[0072] The method for determining that the return address points to the stack space is as follows: during the backtracking process, the return address of each stack frame is checked. If the value of the return address is between the bottom and top of the stack, then the return address points to the stack space.
[0073] Step 4: In order to extract shellcode as accurately as possible, the extraction boundaries need to be confirmed, and the extraction should contain as few "impurities" as possible.
[0074] Determining the upper bound: Starting from the address pointed to by r, scan byte by byte in the direction of decreasing memory, according to priority. Stop scanning if one of the following conditions is encountered:
[0075] P1. At least 5 consecutive 0x00s, with the address of the first 0x00 plus 1 as the upper bound;
[0076] P2. At least 5 consecutive 0xcc addresses, with the address of the first 0xcc address plus 1 as the upper bound;
[0077] P3, at least four consecutive nop instructions (0x90), with the address of the first 0x90 plus 1 as the upper bound; P3 is for cases where some shellcode allocates a large space on the stack and fills it with nop instructions to ensure that the returned address after being tampered with will be hit, such as Figure 3 As shown.
[0078] Figure 3 In this invention, the requirement is not to identify the existence of a slip zone (finding multiple consecutive nop instructions constitutes a slip zone), but to determine the upper bound of the shellcode.
[0079] The "slippery zone" is a potential attack technique. Its principle is that if the attacker isn't confident in precisely locating the shellcode header in memory using a carefully designed pointer, they will fill the memory area before the shellcode header with a large number of nop instructions (no-op instructions that do nothing). Although they can't accurately locate the shellcode header, as long as they locate any part of this large block of nop instructions, the program will execute these nop instructions one by one, eventually reaching the shellcode. This process of executing nop instructions one by one resembles a "slippery" motion. (This can be understood as having poor aim, so they design the target to be larger, so even if they miss, they can still hit it.)
[0080] P4. The upper bound is the address where the value of r is stored plus 4; and the address where the value of r is stored is the top of the stack frame. To ensure integrity, the shellcode will always push the top of the stack frame outside the shellcode entity, such as... Figure 4 As shown.
[0081] Determining the lower bound: Starting from the address pointed to by r, scan byte by byte in the direction of increasing memory size, according to priority. Stop scanning if one of the following conditions is encountered:
[0082] S1. At least 5 consecutive 0x00s, with the address of the first 0x00 minus 1 as the lower bound;
[0083] S2. At least 5 consecutive 0xcc addresses, with the address of the first 0xcc address minus 1 as the lower bound;
[0084] S3, at least 4 consecutive nop instructions (0x90), with the address of the first 0x90 minus 1 as the lower bound;
[0085] S4. If the traceback originates from the ZwTerminateProcess function, then address r is the ending address; this S4 case corresponds to the shellcode failing to repair the stack and choosing to exit the program locally, such as... Figure 5 As shown.
[0086] S5. Perform stack backtracking starting with e_2. If backtracking to the end is successful, use the address where e_2 is stored (i.e., the value of e_1) minus 4 as the lower bound address. This situation corresponds to shellcode being able to automatically repair stack balance, such as... Figure 6 As shown.
[0087] Generally speaking, this range will be larger than the actual shellcode range, including the actual shellcode.
[0088] The purpose of defining this boundary is to minimize the extraction of unnecessary and useless data when extracting shellcode. Continuing to scan will only include more irrelevant data, interfering with the user's process.
[0089] Step 5: Based on the range defined above, extract the binary data, and then use a disassembler, such as ndisasm.exe, to identify it and convert the binary data into x86 assembly code, which will contain the actual shellcode.
[0090] The aforementioned disassemblers refer to programs that can recognize machine code as assembly code. There are many such programs, each with its own steps. These programs are all common, just like the operation of converting images to text, which can be achieved by many means, such as Apple's photo album, WeChat, XunJie software, or by programming yourself.
[0091] The actual shellcode, or "real" shellcode, is the part that is not impurities but is actually needed, as opposed to the redundant and useless data (impurities) mixed in with the extracted data. This extraneous data might be identified as erroneous junk code by disassemblers and thus included in the mix.
[0092] This invention combines the common behavioral characteristics of shellcode and hooks the system API functions that shellcode is most likely to call, thus achieving an intuitive and quick way to locate and extract shellcode. Although the extraction range cannot be completely accurate, it can include the main part of the shellcode with a high probability.
[0093] This invention can also be applied to products that dynamically detect the behavior of virus samples in batches, such as sandboxes. Sandboxes perform batch dynamic detection of virus samples by simultaneously running a large number of potentially virus-laden files sequentially in a virtual machine. During runtime, numerous API functions of the running program are hooked, and the information obtained from these hooks is used to determine whether a file is malicious.
[0094] Because this solution is based on hooks, it can be embedded in the hook code of such sandboxes to determine whether each file contains the attack method of interest. Specifically, it can inject shellcode into the stack space via stack overflow to execute malicious behavior. If the shellcode exists, it is automatically extracted. This saves significant manpower costs.
[0095] While experienced technicians might be able to manually extract shellcode using this method, the debugging process is quite cumbersome. Moreover, when dealing with hundreds of files, it becomes even more difficult to manually extract them one by one, which can easily lead to inefficiency.
[0096] This invention utilizes a stack overflow mechanism to embed shellcode within the stack space. Upon stack overflow, the return address of the stack frame is modified (to point to the shellcode) before execution. Currently, there is no method for extracting shellcode using this approach.
[0097] Although shellcode attacks are not limited to the methods described in this article, the one presented here is one of the most typical and frequently occurring methods.
[0098] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for locating and extracting shellcode from the stack space under Windows x86, characterized in that: This includes step one: hooking the system API functions LdrLoadDll, LdrGetProcedureAddressForCaller, NtCreateUserProcess, and ZwTerminateProcess; Step 2: After a hit, perform stack backtracking on the hooked function. For each stack frame backtracked, check the location pointed to by the return address of that level to see if the location belongs to the stack space. If no return address is found pointing to the stack space after backtracking to the end, the function call is considered safe. Otherwise, it is considered that shellcode is being executed on the stack, and proceed to Step 3. Step 3: When a certain level return address is found to point to the inside of the stack space, record this address value r, and record the EBP value of the nearest level along the address decreasing direction pointed to by r, and record it as e_1. Then, backtrack one level through e_1 to get e_2. At this time, e_2 is the theoretical EBP value of the current stack frame. Step 4: Confirm the boundaries of the extraction; Step 5: Based on the boundary range determined in Step 4, extract the binary data and then use a disassembler to identify it as x86 assembly, in which the actual shellcode will be contained.
2. The method for locating and extracting shellcode in the stack space under Windows x86 according to claim 1, characterized in that: The stack backtracking method in step two is as follows: First, in the hook function of the target function, the value of the EBP register is obtained through inline assembly. The inline assembly is used to accept the value of the EBP register as a DWORD type variable. At this time, the value of EBP is used as a pointer to point to the bottom of the stack frame of the current function. The value stored at the bottom of the stack frame is the value of the EBP register when the previous function is called. This value is used as a pointer, which in turn points to the bottom of the stack frame of the previous function, and so on, so that the bottom of all stack frames of the current thread can be accessed.
3. The method for locating and extracting shellcode in the stack space under Windows x86 according to claim 2, characterized in that: The method to obtain the return address of a function is as follows: Adding 4 to the EBP value creates a pointer that points to the return address of the current function. Similarly, adding 4 to the EBP value of the parent function's stack frame creates a pointer that points to the return address of the parent function.
4. The method for locating and extracting shellcode in the stack space under Windows x86 according to claim 1, characterized in that: The method for determining the stack space in step three is as follows: The top and bottom addresses of the stack space are obtained through the thread environment block of the current thread. The top of the stack is located at an offset of 4 bytes from the thread environment block, and the bottom of the stack is located at an offset of 8 bytes. The two variables, bottom and top, in inline assembly are used to receive the values of the bottom and top of the stack, respectively. The area pointed to by the values between the bottom and top of the stack is the stack space of the current thread.
5. A method for locating and extracting shellcode in the stack space under Windows x86 according to claim 4, characterized in that: The method to determine if the return address points to the stack space is as follows: During the backtracking process, the return address of each stack frame is checked. If the value of the return address is between the bottom and top of the stack, then the return address points to the stack space.
6. The method for locating and extracting shellcode in the stack space under Windows x86 according to claim 1, characterized in that: The boundary in step four is divided into an upper boundary and a lower boundary. The method for determining the upper boundary is as follows: Starting from the address pointed to by r, the system scans byte by byte in the direction of decreasing memory size. When a preset condition is encountered during the upper bound scan, the scan stops, and the corresponding upper bound is determined.
7. A method for locating and extracting shellcode in the stack space under Windows x86 according to claim 6, characterized in that: When a preset condition is encountered during the upper bound scan, the scan is stopped, and the corresponding upper bound is determined, including: P1. At least 5 consecutive 0x00s, with the address of the first 0x00 plus 1 as the upper bound; P2. At least 5 consecutive 0xcc addresses, with the address of the first 0xcc address plus 1 as the upper bound; P3, at least 4 consecutive nop instructions (0x90), with the address of the first 0x90 plus 1 as the upper bound; P4. The upper bound is the address where the r value is stored plus 4.
8. A method for locating and extracting shellcode in the stack space under Windows x86 according to claim 6, characterized in that: The method for determining the lower bound is as follows: Starting from the address pointed to by r, the system scans byte by byte in the direction of increasing memory size. When a preset condition is encountered during the lower bound scan, the scan stops, and the corresponding lower bound is determined.
9. A method for locating and extracting shellcode in the stack space under Windows x86 according to claim 8, characterized in that: When a preset condition is encountered during the lower bound scan, the scan is stopped, and the corresponding lower bound is determined, including: S1. At least 5 consecutive 0x00s, with the address of the first 0x00 minus 1 as the lower bound; S2. At least 5 consecutive 0xcc addresses, with the address of the first 0xcc address minus 1 as the lower bound; S3, at least 4 consecutive nop instructions (0x90), with the address of the first 0x90 minus 1 as the lower bound; S4. If it traces back from the ZwTerminateProcess function, then the address r is the ending address; S5. Perform stack backtracking with e_2 as the initial value. If the backtracking is successful, the lower bound address is the address where e_2 is stored minus 4.
Citation Information
Patent Citations
A shellcode detection method and device
CN104008336B
Method, device and equipment for defending against network attacks
CN101621498A
Method and apparatus for detecting Shellcode based on stack frame abnormity
CN105678168A