JAVA Deserialization Exploit Attack Detection

The method employs a JAVA virtual machine to detect and prevent input deserialization exploit attacks by analyzing stack traces and issuing alerts or preventing malicious execution, effectively addressing the lack of effective detection mechanisms in existing systems.

JP7661534B2Active Publication Date: 2025-04-14PALO ALTO NETWORKS INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023572867
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-05-27
Publication Date
2025-04-14
Estimated Expiration
2041-05-27

AI Technical Summary

Technical Problem

Existing computer systems lack effective mechanisms to detect and prevent input deserialization exploit attacks, which can lead to malicious software execution and security breaches.

Method used

A method and system that utilize a JAVA virtual machine (JVM) to receive and deserialize input data streams, detect software processes initiated in response to deserialization, and issue alerts or prevent malicious execution.

Benefits of technology

The solution effectively detects and mitigates input deserialization exploit attacks by analyzing stack traces and comparing function calls with predefined lists, thereby enhancing the security of computer systems against such threats.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007661534000006
    Figure 0007661534000006
  • Figure 0007661534000007
    Figure 0007661534000007
  • Figure 0007661534000008
    Figure 0007661534000008
Patent Text Reader

Abstract

A method, apparatus, and computing program product implements an embodiment of the invention for protecting a computer system, comprising receiving, by a processor executing a JAVA virtual machine, an input including a serialized data stream, deserializing the received serialized data stream by the JAVA virtual machine, and upon detecting a software process initiated in response to deserializing the data stream, an alert is issued regarding the detected software process.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate generally to computer security, and more particularly to detecting input deserialization exploit attacks. [Background technology]

[0002] Many computer and network systems are deployed with multiple layers of security devices and software to detect and repel an ever-growing range of security threats. At the most basic level, computers use anti-virus software to prevent malicious software from executing on the computer. At the network level, intrusion detection and prevention systems analyze and control network traffic to detect and prevent malware from spreading through the network.

[0003] The above discussion is presented as a general overview of the related art in this field, and should not be construed as an admission that any of the information it contains constitutes prior art to the present patent application. Summary of the Invention

[0004] According to one embodiment of the invention, a method for protecting a computer system may be provided that includes the steps of receiving, by a processor executing a JAVA Virtual Machine (JVM), an input including a serialized data stream, deserializing, by the JVM, the received serialized data stream, detecting a software process initiated in response to deserializing the data stream, and issuing an alert regarding the detected software process.

[0005] In one embodiment, the input includes a hypertext transfer protocol request.

[0006] In another embodiment, detecting a software process initiated in response to deserializing the data stream includes analyzing a stack trace including one or more functions, where a given function initiated the software process and detecting the deserialization of the data stream in the stack trace.

[0007] In an additional embodiment, the stack trace includes a set of stack frames, a first given stack frame includes the function that initiated the software process and a second given stack frame includes the deserialization of the data stream, and the step of detecting a software process initiated in response to deserializing the data stream includes detecting that the first given stack frame and the second given stack frame are within a specified number of stack frames.

[0008] In a further embodiment, the step of detecting a software process initiated in response to deserializing the data stream includes comparing the one or more functions in the stack trace to a list of first functions and detecting a match between any given function in the stack trace and any given first function.

[0009] In another embodiment, the step of detecting a software process initiated in response to deserializing the data stream includes the steps of comparing the one or more functions in the stack trace to a list of second functions, detecting a match between any given function in the stack trace and any given second function, and not detecting a match between any given function in the stack trace and any first function.

[0010] In a supplemental embodiment, issuing the alert includes preventing execution of the software process.

[0011] According to one embodiment of the present invention, there is also provided a computer system including a memory configured to store a JAVA Virtual Machine (JVM) and a processor configured to receive an input including a serialized data stream, deserialize, by the JVM, the received serialized data stream, detect software processes initiated in response to deserializing the data stream, and issue an alert regarding the detected software processes.

[0012] According to one embodiment of the present invention, there is additionally provided a computer software product for protecting a computer system, the product including a non-transitory computer readable medium having stored thereon program instructions which, when read by a computer, cause the computer to execute a JAVA Virtual Machine (JVM), receive input including a serialized data stream, deserialize, by the JVM, the received serialized data stream, detect software processes initiated in response to deserializing the data stream, and issue an alert regarding the detected software processes.

[0013] In accordance with one embodiment of the present invention, there is further provided a method for protecting a computer system, the method including the steps of receiving, by a processor executing a JAVA Virtual Machine (JVM), an input including a serialized data stream, deserializing, by the JVM, the received serialized data stream, detecting software libraries loaded into memory in response to deserializing the data stream, and issuing an alert regarding the detected software libraries.

[0014] In one embodiment, the input includes a hypertext transfer protocol request.

[0015] In another embodiment, detecting a software process initiated in response to deserializing the data stream includes analyzing a stack trace including one or more functions, where a given function loads the software library into the memory, and detecting the deserialization of the data stream in the stack trace.

[0016] In an additional embodiment, the stack trace includes a set of stack frames, a first given stack frame includes the function that loaded the software library into memory, and a second given stack frame includes the deserialization of the data stream, and the step of detecting the software library loaded into memory in response to deserializing the data stream includes detecting that the first given stack frame and the second given stack frame are within a specified number of stack frames.

[0017] In a further embodiment, detecting a software process initiated in response to deserializing the data stream includes comparing the one or more functions in the stack trace to a list of first functions and detecting a match between any given function in the stack trace and any given first function.

[0018] In a supplemental embodiment, detecting a software process initiated in response to deserializing the data stream includes comparing the one or more functions in the stack trace to a list of second functions, detecting a match between any given function in the stack trace and any given second function, and not detecting a match between any given function in the stack trace and any first function.

[0019] In some embodiments, issuing the alert includes preventing the software library from being loaded into memory. [Brief description of the drawings]

[0020] The present disclosure is herein described, by way of example only, with reference to the accompanying drawings. [Figure 1] FIG. 1 is a schematic block diagram of a server computer executing a hook function configured to detect input deserialization exploit attacks in accordance with one embodiment of the present invention. [Diagram 2] FIG. 2 is a flow chart that generally illustrates a method for initializing a server to detect input deserialization exploit attacks, according to one embodiment of the present invention. [Diagram 3] FIG. 3 is a flow chart that outlines the steps of a deserialization exploit attack according to one embodiment of the present invention. [Figure 4] FIG. 4 is a flow chart that illustrates generally a method for detecting an input deserialization exploit attack involving a malicious process launched in response to a deserialized data stream according to a first embodiment of the present invention. [Diagram 5] FIG. 5 is a flow chart that generally illustrates a method for detecting an input deserialization exploit attack involving a malicious software library loaded in response to a deserialized data stream, according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Software environments such as JAVA (produced by Oracle Corporation, 500 Oracle Parkway Redwood City, CA 94065 USA) use serialization and deserialization to manage objects on computing devices. A serialization operation takes an in-memory data structure (e.g., an object) and converts it into a data stream that can be stored and / or transmitted. A deserialization operation takes the data stream and reconstructs it into a copy of the in-memory data structure. For example, a first computer can serialize user input into a data stream and then send the data stream to a second computer. Upon receiving the data stream, the second computer can deserialize the data stream into a copy of the user input.

[0022] Some cyber attacks exploit weaknesses in serialization / deserialization to transmit malicious serialized data streams to web servers and other types of computers connected to public networks, such as the Internet. For example, an attacker can send a serialized file that represents a malicious payload, which when deserialized can trigger the execution of the attacker's code, which can perform malicious actions, such as Denial of Service (DoS) attacks and reverse shell spawning.

[0023]

[0006] An embodiment of the present invention provides a method and system for protecting a computer system from attacks that utilize serialization and deserialization. As described herein below, a processor executing a JAVA Virtual Machine (JVM) receives an input that includes a serialized data stream. Upon deserializing the serialized data stream and detecting a software process that was initiated in response to deserializing the data stream, an alert is issued regarding the detected software process.

[0024] In embodiments described herein, software processes that are started in response to a deserialized data stream can be detected by analyzing the stack trace of the function that starts the software process. In some embodiments, these functions can be modified (i.e., "hooked") to analyze their respective stack traces when called.

[0025] In addition to detecting processes initiated in response to a deserialized data stream, embodiments of the present invention can be used to detect other types of cyber-attacks initiated by malicious payloads embedded in serialized input. For example, a cyber-attack may include a malicious software library that is loaded in response to deserializing a data stream.

[0026] SYSTEM DESCRIPTION Figure 1 is a schematic block diagram of an example of a server 20 executing a hook function 22 configured to detect processes 24 initiated in response to a deserialized data stream 26, in accordance with one embodiment of the present invention. In the configuration shown in Figure 1, the server 20 communicates with a client computer 28 via a public network 30, such as the Internet.

[0027] In the embodiments described herein, a client computer 28 can launch an attack against the server 20 by embedding malicious instructions into a serialized data stream 32 that the client computer communicates to the server 20 over the Internet 30. In one embodiment, the client computer 28 can execute a client application 34 that includes an input screen 36, and a user (not shown) can enter malicious instructions into the input screen. Upon receiving the malicious instructions via the input screen, the client application 34 can serialize the malicious instructions into the serialized data stream 32 and communicate the serialized data stream to the server 20 over the Internet 30.

[0028] In another embodiment, the client computer 28 may serialize the malicious instructions into a HyperText Transfer Protocol (HTTP) request 38 and communicate the HTTP request over the Internet 30 to the server 20. Upon receipt, the server 20 stores the HTTP request 38 as a serialized data stream 32.

[0029] The server 20 comprises a processor 40, a memory 42, and a storage device 44. In the configuration shown in Figure 1, the memory 42 may store: · A JAVA Virtual Machine (JVM) 48 which executes code in response to the de-serialized data stream 26. In the configuration shown in Figure 1, the JVM 48 runs on the processor 40. A plurality of software processes 24 that are executed by a processor 40 from memory. One or more software libraries 50 (e.g., MyLibrary.so) that the processor 40 can execute from memory. In one embodiment, the processor 40 can load a given software library 50 into memory (e.g., from storage device 44). In another embodiment, the processor 40 can retrieve a given software library from a client computer 28 (i.e., via the Internet 30) and then load the retrieved software library into memory 42.

[0030] Typically, functions 22 and 52 (as well as any of the functions in whitelist 50 and blacklist 62) are loaded by processor 40 from library 50. In some embodiments, the name of a given function may have the format LibraryName.FunctionName, where LibraryName contains the name of the given library 50 and FunctionName contains the name of the given function. The functions 52 are executable by the processor 40 from memory. In the embodiment described herein, at least one given function 52 includes a hook function call 54 to a hook function 22 that the processor can also execute from memory. Any given function 52 that includes a given hook function call 54 is referred to herein as a hooked function 52. Hook function calls 54 and hook functions 22 are described below. The stack traces 56 include a respective set of stack frames 57. In some embodiments, each given invocation of each given function 52 has a corresponding stack trace 56. Each stack frame in a given stack trace stores information about all invocations (i.e., of additional functions 52) from the start of the thread to the given invocation of the given function 52. An example of a given stack trace is presented and explained in the description below with reference to FIG. 4. The whitelist 58 includes a set of whitelisted functions 60. A given whitelisted function 60 typically includes the name of a given function 52. The whitelist 58 is described in the description below with reference to Figures 2, 4, and 5. The blacklist 62 includes a set of blacklisted functions 64. In an embodiment of the invention, each blacklisted function 64 typically includes the name of a given function 52 that deserializes the serialized data stream 32. The blacklist 62 is explained in the description below with reference to Figures 2, 4, and 5.

[0031] A given hook function call 54 includes modifications to a given function 52 that enable the given function to call the hook function 22 upon invocation of the given function. In an embodiment described below, the hook function 22 analyzes a stack trace corresponding to a call of the given function to determine if the given function was called in response to deserializing the serialized data stream 32. For example, if the given function creates a new process 24, then upon invocation of the given function, the hook function call within the given function calls the hook function 22 to determine if the processor 40 created a new process in response to the deserialized data stream 26.

[0032] Below is an example of pseudocode for a given hook function 52 (line numbers inserted for clarity): [Table 1]

[0033] In this example, the hooked function starts a new process only upon determining that a call to the hooked function is not responsive to deserializing an input, such as serialized data stream 32.

[0034] In some embodiments, the tasks involved in deserializing serialized data stream 32 and executing deserialized data stream 26 and functions 22 and 52 may be performed by one or more virtual machines running on one or more instances of server 20 or may be performed by one or more cloud-based applications.

[0035] The processor 40 comprises a general-purpose central processing unit (CPU) or a dedicated embedded processor, which is programmed with software or firmware to perform the functions described herein. The software may be downloaded to the server 20 in electronic form, for example, over a network. Additionally or alternatively, the software may be stored on a tangible, non-transitory computer-readable medium, such as an optical, magnetic, or electronic memory medium. Further, additionally or alternatively, at least some of the functions of the processor 40 may be performed by hard-wired or programmable digital logic circuitry.

[0036] Examples of memory 42 include dynamic random access memory and non-volatile random access memory. Examples of storage device 44 include non-volatile storage devices, such as hard disk drives and solid state disk drives.

[0037] EXPOLIT DETECTION 2 is a flow chart that generally illustrates a method for initializing server 20 to detect deserialization exploits, according to one embodiment of the present invention. In step 70, a software application developer (not shown) codes hook functions 22 and inserts hook function calls 54 into one or more functions 52. In the embodiment described herein, hook functions 52 typically perform sensitive operations, such as creating a new process 24 or loading a given software library 50.

[0038] In step 72, a system administrator (not shown) initializes the whitelist 58 by adding one or more whitelisted functions 60. One example of a given whitelist function 60 is as follows: [Table 2]

[0039] In step 74, a system administrator initializes the blacklist 62 by adding one or more blacklisted functions 64. As mentioned above, a given function 60 or 64 typically includes the name of a given function 52. Examples of blacklisted functions 64 include, but are not limited to, JSONdeserializerImpl, ElementHandler, ObjectInputStream, and SimpleNode!getValue.

[0040] Finally, in step 76, the system administrator activates the functionality of the hooked functions, the whitelist 58, and the blacklist 62 on the server 20. Note that at any point after activating the whitelist 58 and blacklist 62, the system administrator can add new functions 60 or 64 or remove given functions 60 or 64, as desired.

[0041] FIG. 3 is a flow chart that generally illustrates a method for performing a non-serialization exploit attack, according to one embodiment of the present invention.

[0042] In step 80, an attacker (not shown) embeds a malicious data stream into the input. In some embodiments, the malicious data stream may include software instructions, such as JAVA code.

[0043] As mentioned above, an attacker can input a malicious data stream into the input screen 36 or embed a malicious data stream into the input including the HTTP request 38. In the embodiments described herein, the malicious data stream can instruct the server 20 to execute a given process 24 (also referred to herein as a malicious process) that may be malicious, or can instruct the server 20 to load a given software library 50 (also referred to herein as a malicious software library) that may be malicious.

[0044] Below is one example of a malicious data stream that an attacker can embed into the input, whose "goal" is to cause the server 20 to execute a given malicious process 24 (also referred to herein as the malicious process): [Table 3]

[0045] For purposes of clarity, each line of the data stream in this example is numbered. In this example, line #5 instructs the server to begin executing the malicious process.

[0046] Below is an example of a malicious data stream commanding a server to load a given malicious software library 50: [Table 4]

[0047] In step 82 , the client computer 28 serializes the input (ie, including the malicious data stream) into the serialized data stream 32 .

[0048] In step 84 , the client computer 28 communicates the serialized data stream 32 to the server 20 over, for example, the Internet 30 .

[0049] In step 86 , processor 40 receives an input including serialized data stream 32 and stores the serialized data stream in memory 42 .

[0050] In step 88 , JVM 48 (running on processor 40 ) deserializes serialized data stream 32 into unserialized data stream 26 .

[0051] Finally, in step 90, in response to deserializing the serialized data stream 32, the JVM 48 (running on the processor 40) begins execution of the malicious data stream embedded in the deserialized data stream 26. As described above, the malicious data stream may instruct a library of the processor 40 (i.e., in response to deserializing the serialized data stream 32) to begin execution of a given malicious process 24 or to load malicious software. Methods for detecting execution of a malicious data stream are described below in the description with reference to Figures 4 and 5.

[0052] FIG. 4 is a flow chart that generally illustrates a method for detecting a given process 24 that a processor 40 has initiated (i.e., its execution) in response to deserializing a JVM's serialized data stream 32, in accordance with one embodiment of the present invention.

[0053] In step 100, a given hooked function 52 is invoked and starts the execution of a new process 24. In the embodiment described in the description with reference to the flowchart presented in Figure 4, the hooked function is invoked (i.e., called) to start any new process 24.

[0054] In step 102, the given hooked function calls the hook function 22, via the hook function call 54, to identify the function in the stack trace that corresponds to a call of the given hooked function.

[0055] Below is an example of a given stack trace 56 that contains multiple stack trace frames 57. [Table 5]

[0056] For purposes of clarity, each stack frame 57 in this example is numbered.

[0057] In step 104, hook function 22 compares the (names of) the functions in the corresponding stack trace to blacklisted functions 64. As described above, blacklisted functions 64 reference functions 52 that can deserialize serialized data stream 32. In an embodiment of the invention, hook function 22 analyzes the stack trace to determine whether processor 40 started a given process 24 in response to deserializing serialized data stream 32.

[0058] In one embodiment, the hook function 22 may determine that the processor 40 started a given process 24 in response to deserializing the serialized data stream 32 by detecting a match between any of the blacklisted functions and any of the functions in the stack frame prior to the invocation of the hooked function. In another embodiment, the hook function 22 may determine that the processor 40 started a given process 24 in response to deserializing the serialized data stream 32 by detecting a match between any of the blacklisted functions and any of the functions in a specified number of stack frames (e.g., 20, 25, or 30) prior to the invocation of the hook function that started the given process.

[0059] In the example stack trace listed above, stack frame #2 shows the invocation of the (hooked) function "ProcessBuilder", and stack frame #19 shows the deserialization of a given serialized data stream 32. Thus, the deserialization operation was performed 17 before the invocation of the given function that started the given process.

[0060] If, in step 106, the hook function 22 detects a match between any of the blacklisted functions and any of the functions in the stack frame prior to the invocation of the hooked function, then, in step 108, the hook function compares the identified name of the function in the stack frame with the whitelisted functions 60.

[0061] If, at step 110, the hook function 22 does not find a match between any of the identified (function names) in the stack frame and any of the whitelisted functions 60, then, at step 112, the hook function (or the hooked function) issues an alert regarding the new process (i.e., the process started by the processor 40 at step 100) and the method ends. In some embodiments, the hook function 22 may issue the alert by preventing the execution of the new process (or terminating execution of a given process 24 that launches the new process). Additionally or alternatively, the hook function 22 may issue the alert by communicating a warning message to a system administrator of the server 20.

[0062] Returning to step 110, if the hook function 22 finds a match between the identified (name of the function) in the stack frame and the whitelisted function 60, the method ends. In other words, if there is a match between any of the functions in the stack frame and any of the blacklisted functions 64, and there is a match between the identified function in the stack frame and any of the whitelisted functions 60, then the match against the whitelisted function "exonerates" (i.e., removes suspicion from) the match against the blacklisted function 64.

[0063] Returning to step 106, if the hook function 22 does not find a match between any of the blacklisted functions and any of the functions in the stack frame prior to the call to the hooked function, the method ends.

[0064] FIG. 5 is a flow chart that generally illustrates a method for detecting a given software library 50 that a processor 40 has loaded into memory 42 in response to deserializing a JVM's serialized data stream 32, in accordance with one embodiment of the present invention.

[0065] In step 120, the given hooked function 52 is invoked, which loads the given software library 50 into memory 42. In the embodiment described in the description with reference to the flowchart presented in FIG. 4, the hooked function is invoked (i.e., called) to load the given library 59.

[0066] In step 102, the hook function identifies the function in the stack trace that corresponds to a call to a given hooked function.

[0067] In step 124, hook function 22 compares the (names of) the functions in the corresponding stack trace to blacklisted functions 64. As described above, blacklisted functions 64 reference functions 52 that can deserialize serialized data stream 32. In an embodiment of the present invention, hook function 22 analyzes the stack trace to determine whether processor 40 loaded a given software library 50 in response to deserializing serialized data stream 32.

[0068] In one embodiment, hook function 22 may determine that processor 40 loaded a given software library 50 in response to deserializing serialized data stream 32 by detecting a match between any of the blacklisted functions and any of the functions in the stack frame prior to the invocation of the hooked function. In another embodiment, hook function 22 may determine that processor 40 loaded a given software library 50 in response to deserializing serialized data stream 32 by detecting a match between any of the blacklisted functions and any of the functions in a specified number (e.g., 20, 25, or 30) of stack frames prior to the invocation of the hook function that loaded the given software library.

[0069] If, in step 126, the hook function 22 detects a match between any of the blacklisted functions and any of the functions in the stack frame prior to the invocation of the hooked function, then, in step 128, the hook function compares the identified name of the function in the stack frame with the whitelisted functions 60.

[0070] If, in step 130, the hook function 22 does not find a match between any of the identified (function names) in the stack frame and any of the whitelisted functions 60, then, in step 132, the hook function (or hooked function) issues an alert regarding the loaded library (i.e., the library loaded by the processor 40 in step 120) and the method ends. In some embodiments, the hook function 22 can issue an alert by preventing the loading of any library 50 into memory 42 (i.e., by the hooked function). For example, the hook function 22 can terminate the execution of a given process 24 that loads any given library. Additionally or alternatively, the hook function 22 can issue an alert by communicating a warning message to a system administrator of the server 20.

[0071] Returning to step 130, if the hook function 22 finds a match between the identified (name of the function) in the stack frame and the whitelisted function 60, the method ends. In other words, if there is a match between any of the functions in the stack frame and any of the blacklisted functions 64, and there is a match between the identified function in the stack frame and any of the whitelisted functions 60, then the match against the whitelisted function "rules out" (i.e., removes suspicion from) the match against the blacklisted function 64.

[0072] Returning to step 126, if the hook function 22 does not find a match between any of the blacklisted functions and any of the functions in the stack frame prior to the call to the hooked function, the method ends.

[0073] As mentioned above, the name of a given function may have the format LibraryName.FunctionName, where LibraryName includes the name (i.e., an identifier or prefix) of the given library 50 and FunctionName includes the name of the given function. In some embodiments, a given blacklisted function 64 or a given whitelisted function 60 may include a library name and / or a function name. In these embodiments, when comparing functions in a stack trace to whitelisted / blacklisted functions, hook function 22 may compare all (i.e., the library name and function name) or a portion (i.e., the library name or the function name) of the stack trace function to the whitelisted / blacklisted functions.

[0074] It will be understood that the above-described embodiments are cited by way of example, and that the present invention is not limited to what has been particularly shown and described above. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described above, as well as variations and modifications thereof, which would occur to those skilled in the art upon reading the foregoing description and which have not been disclosed in the prior art.

Claims

1. 1. A method for securing a computer system, comprising: receiving, by a processor executing a Java Virtual Machine (JVM), an input comprising a serialized data stream; deserializing, by the JVM, the received serialized data stream; invoking, by the JVM, a hook function in response to deserializing the serialized data stream; analyzing a stack trace by the hook function, the stack trace comprising a set of additional functions corresponding to the hook function and within a thread for the hook function; detecting, in the stack trace, a software process that was initiated in response to deserializing the data stream; issuing an alert regarding the detected software process; A method comprising:

2. the input includes a hypertext transfer protocol request; The method of claim 1.

3. the stack trace includes a set of stack frames; the first given stack frame includes the hook function that started the software process; a second given stack frame includes the deserialization of the data stream; and the step of detecting a software process initiated in response to deserializing the data stream includes detecting that the first given stack frame and the second given stack frame are within a specified number of stack frames. The method of claim 1.

4. The step of detecting a software process initiated in response to deserializing the data stream comprises: comparing one or more additional functions in the stack trace to the first list of functions; detecting a match between any given additional function in the stack trace and any given first function; The method of claim 3 , comprising:

5. The step of detecting a software process initiated in response to deserializing the data stream comprises: comparing one or more additional functions in the stack trace to a second list of functions; detecting a match between any given additional function in the stack trace and any given second function; not detecting a match between any given additional function in the stack trace and any first function; The method of claim 4 , comprising:

6. The step of issuing the alert comprises: preventing execution of said software process; The method of any one of claims 1 to 5, comprising:

7. 1. A computer system comprising: a memory configured to store a Java Virtual Machine (JVM); a processor; Receives an input including a serialized data stream; deserializing, by the JVM, the received serialized data stream; invoking, by the JVM, a hook function in response to deserializing the serialized data stream; analyzing a stack trace with the hook function, the stack trace corresponding to the hook function and comprising a set of additional functions within a thread for the hook function; Finding, in the stack trace, a software process that was initiated in response to deserializing the data stream; and issuing an alert regarding the detected software process; a processor configured to 2. A computer system comprising:

8. the input includes a hypertext transfer protocol request; 8. The computer system of claim 7.

9. the stack trace includes a set of stack frames; the first given stack frame includes the hook function that started the software process; a second given stack frame includes the deserialization of the data stream; and the processor is configured to detect the software process that was initiated in response to deserializing the data stream by detecting that the first given stack frame and the second given stack frame are within a specified number of stack frames.

8. The computer system of claim 7.

10. The processor, configured to detect the software process that was started in response to deserializing the data stream by comparing one or more additional functions in the stack trace to a list of first functions; and finding a match between any given additional function in the stack trace and any given first function; 8. The computer system of claim 7.

11. The processor, and configured to detect software processes launched in response to deserializing the data stream by comparing one or more additional functions in the stack trace to a second list of functions; Detecting a match between any given additional function in the stack trace and any given second function; and not finding a match between any given additional function in the stack trace and any given first function; 11. The computer system of claim 10.

12. The processor, and issuing the alert by preventing execution of the software process. A computer system according to any one of claims 7 to 11.

13. 1. A computer program for protecting a computer system, the computer program comprising a plurality of instructions and stored on a non-transitory computer readable medium, The instructions, when read by a computer, cause the computer to: Run the Java Virtual Machine (JVM), Receives an input including a serialized data stream; deserializing, by the JVM, the received serialized data stream; invoking, by the JVM, a hook function in response to deserializing the serialized data stream; analyzing a stack trace with the hook function, the stack trace corresponding to the hook function and comprising a set of additional functions within a thread for the hook function; Finding, in the stack trace, a software process that was initiated in response to deserializing the data stream; and issuing an alert regarding the detected software process; A computer program that causes

14. 1. A method for securing a computer system, comprising: receiving, by a processor executing a Java Virtual Machine (JVM), an input comprising a serialized data stream; deserializing, by the JVM, the received serialized data stream; invoking, by the JVM, a hook function in response to deserializing the serialized data stream; analyzing a stack trace by the hook function, the stack trace comprising a set of additional functions corresponding to the hook function and within a thread for the hook function; detecting, in the stack trace, software libraries that were loaded into memory in response to deserializing the data stream; issuing an alert regarding the detected software libraries; A method comprising:

15. the input includes a hypertext transfer protocol request; The method of claim 14.

16. the stack trace includes a set of stack frames; the first given stack frame includes the hook function that loaded the software library into memory; a second given stack frame includes the deserialization of the data stream; and and detecting the software library loaded into memory in response to deserializing the data stream includes detecting that the first given stack frame and the second given stack frame are within a specified number of stack frames. The method of claim 14.

17. The step of detecting the software library initiated in response to deserializing the data stream comprises: comparing one or more additional functions in the stack trace to the first list of functions; detecting a match between any given additional function in the stack trace and any given first function; 15. The method of claim 14, comprising:

18. The step of detecting the software library initiated in response to deserializing the data stream comprises: comparing one or more additional functions in the stack trace to a second list of functions; detecting a match between any given additional function in the stack trace and any given second function; not detecting a match between any given additional function in the stack trace and any first function; 20. The method of claim 17, comprising:

19. The step of issuing the alert comprises: preventing the software library from being loaded into memory; 19. The method of any one of claims 14 to 18, comprising:

Citation Information

Patent Citations

  • How to verify an API using information recorded in the call stack

    JP2010509654A

  • Aggression detection device and method using vulnerable point of program

    JP2015141718A