A stateful protocol fuzz testing method, medium, and product based on reinforcement learning

Abstract

This invention discloses a fuzz testing method, medium, and product for stateful protocols based on reinforcement learning. The method includes inputting the network traffic of the test target into a state model containing packet information, extracting seed sequence mutations to generate test cases, including evaluating candidate states of the test target and selecting a target state, obtaining a packet sequence guiding to the target state, and performing segmented mutation to obtain multiple segments; the fuzzer synchronously updates the sampling probability of the segments based on the cumulative number of reward values, selecting segments with sampling probabilities exceeding a threshold to generate test cases; when a new state of the test target is triggered during testing, the new state and the triggering packet are added to the state model containing packet information, and the packet sequence that triggered the new state is added as a new seed to the seed pool for updating. This invention effectively improves the targeting and efficiency of fuzz testing.

Description

Technical Field

[0001] This invention relates to a fuzz testing method, medium, and product for stateful protocols based on reinforcement learning, belonging to the field of network protocol fuzz testing technology. Background Technology

[0002] Fuzzing techniques are widely used in network protocol vulnerability discovery. The core of fuzzing network protocols is to automatically or semi-automatically construct abnormal message data and send it to the network communication program, while monitoring the program's operational status (crashes, service interruptions, etc.) in real time to discover security vulnerabilities. Based on the different methods of test case construction, protocol fuzzing techniques can be divided into two categories: generation-based fuzzing and mutation-based fuzzing.

[0003] In generative fuzzing, test case construction relies on a protocol description model provided by the tester. Before implementing fuzzing, testers need to expend considerable effort to obtain information such as the protocol format and state machine. Although automated protocol reverse engineering tools exist to reduce the difficulty of protocol analysis, the accuracy of the analysis is insufficient to meet the requirements of fuzzing.

[0004] Mutation-based fuzzing methods mutate captured network packets to construct test cases, eliminating the need for testers to analyze the protocol. When fuzzing stateful protocols, mutation-based fuzzing methods often use bootstrap messages to guide the protocol program to a specific state before mutating subsequent packets to conduct tests.

[0005] Depending on the information required, fuzzing can be categorized into gray-box fuzzing and black-box fuzzing. Existing gray-box fuzzing methods often employ source code instrumentation to obtain internal information about the target program, tracking memory allocation and network I / O flows through stub feedback. While these methods can accurately understand the protocol state by obtaining internal feedback during program execution, they also have limitations. First, existing methods mostly utilize compile-time instrumentation; if the target program's source code is unavailable, these methods are difficult to apply. Second, instrumentation methods require maintaining probe data in memory, and state restoration consumes resources, making them difficult to apply under resource-constrained conditions. In contrast, black-box fuzzing does not rely on source code; the tester only needs to send test cases to the program under test and monitor its performance after processing the test cases, making it more widely applicable.

[0006] Existing mutation-based protocol black-box fuzzing methods suffer from the following main problems: First, message mutation typically employs sequential or randomized operations. These operations are inherently blind, failing to focus mutations on key fields to ensure that mutated test cases explore a broader state space. Second, state selection is also blind. Existing methods often use random or rule-based approaches for state selection. Random strategies make it difficult to focus fuzzing on states with in-depth testing value. Fixed-rule testing, on the other hand, can easily lead to local optima in state selection. Furthermore, most fuzzing methods do not fully utilize the experience accumulated during the fuzzing process to improve efficiency. These factors constrain the overall performance of fuzzing.

[0007] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0008] The technical problem to be solved by this invention is: how to improve the targeting and efficiency of fuzz testing.

[0009] To achieve the above objectives / to solve the above technical problems, the present invention is implemented using the following technical solution.

[0010] In a first aspect, the present invention provides a stateful protocol fuzzing method based on reinforcement learning, comprising:

[0011] Capture network traffic generated by the test target during communication;

[0012] The network traffic generated by the test target during communication is input into a state model containing message information. A seed sequence including state information and message content is extracted and the seed sequence is output.

[0013] The seed sequence is mutated to generate test cases for testing the target.

[0014] The method for generating test cases by mutating the seed sequence includes:

[0015] The seed sequence is input into the fuzzer to evaluate the candidate states of the test target and select the target state to be tested from the candidate states;

[0016] By sending the pilot message in the seed sequence, the test target is guided to the target state;

[0017] The message corresponding to the selected target state is divided into multiple segments according to special symbols;

[0018] Based on the Thompson sampling algorithm, a segment is selected from the multiple segments to perform segment mutation, resulting in a mutated segment and an unmutated segment. The mutated segment and the unmutated segment are then concatenated to generate test cases.

[0019] Send the test cases to the test target for testing, and repeat the following steps until the preset test termination condition is met:

[0020] When the state of the test target changes, the reward mechanism is triggered, and the reward value is accumulated.

[0021] The sampling probability of the segment that mutated in the current step is updated based on the cumulative reward value using a fuzzer.

[0022] Select the segment with the highest sampling probability for mutation, and then combine the mutated segment and the unmutated segment in the current step to construct a new test case.

[0023] Send new test cases to the test target for testing and monitor changes in the test target's status.

[0024] Specifically, when the state of the test target changes to a new state, the new state of the test target and the trigger message are added to the state model containing message information for updating, and the message sequence that triggers the new state is added to the seed pool as a new seed sequence.

[0025] Furthermore, the method for constructing the state model containing message information includes:

[0026] Capture the network traffic generated by the test target during each communication, and extract data packets from the network traffic;

[0027] Based on the source IP address and destination IP address of the data packet, the data packet is divided into server response messages and client service request messages;

[0028] Extract keywords from the server response message to represent the server's status and establish an initial status model based on the keywords. The keywords include status codes and text content.

[0029] Extract the application layer payload from the client service request message, perform mutation processing to obtain the mutated payload, encapsulate the mutated payload back into the client service request message data packet, and send the encapsulated client service request message data packet to the server for testing;

[0030] The client service request message and the initial state model are merged to obtain a state model containing message information.

[0031] Furthermore, the initial state model and the state model containing message information are stored in a graph structure. Each node in the graph structure represents a state of the test target, and the edges between nodes represent messages that cause the test target state to change. When a new state appears in the test target, the new state is inserted into the graph structure as a new node.

[0032] Furthermore, the method for selecting the target state includes:

[0033] The UCB algorithm is used to evaluate the candidate states of the test target and calculate the UCB value of each candidate state.

[0034] The candidate state with the highest UCB value is taken as the target state for the current test.

[0035] Furthermore, the method of segmenting the message corresponding to the selected target state into multiple segments according to special symbols includes:

[0036] If the message contains special symbols, the message is divided into multiple segments based on the special symbols.

[0037] When the message does not contain special symbols, the message is divided into multiple segments using a fixed-length segmentation method.

[0038] Furthermore, the rules of the reward mechanism are as follows:

[0039] When the test target changes to a new state, the reward value of the segment that mutated in the current step is increased by "1";

[0040] When the test target changes to a known state, the reward value of the segment that mutated in the current step is increased by "0".

[0041] Furthermore, the rules of the reward mechanism also include:

[0042] When the test target enters an abnormal handling state, the reward value of the segment that mutated in the current step is incremented by "0";

[0043] When communicating via a client / server architecture protocol, the reward value of the segment that has mutated in the current step is dynamically updated based on the server's real-time response to the client's request.

[0044] Furthermore, the method for accumulating the reward value includes:

[0045] Repeat the following steps until the number of tests for the message corresponding to the currently selected target state reaches the test termination condition:

[0046] Record the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0";

[0047] Based on the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0", the sampling probability of the segment in the message that has mutated is calculated using the β distribution probability density function.

[0048] In a second aspect, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described in the first aspect.

[0049] Thirdly, the present invention provides a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the method described in the first aspect.

[0050] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0051] 1. This invention proposes an innovative reinforcement learning-based fuzzing method for stateful protocols, deeply integrating reinforcement learning theory with fuzzing techniques. By introducing a reinforcement learning mechanism and combining a state selection strategy with the segmented mutation technique of the Thompson sampling algorithm, the intelligence and efficiency of stateful protocol fuzzing are significantly improved.

[0052] 2. The method provided by this invention not only guides testing to cover new states and untested known states, but also provides feedback on test performance through a reward mechanism, enabling precise mutation and optimization of test cases. This invention significantly reduces the generation of redundant test cases, making the fuzzing process more efficient and targeted.

[0053] 3. Furthermore, this invention can dynamically update the state model and expand the seed pool, ensuring comprehensive test coverage and effectively improving the depth and breadth of vulnerability discovery, providing strong technical support for vulnerability mining. Simultaneously, by dynamically adjusting the test strategy based on the state transitions of the test target, continuous optimization of the testing process is achieved, providing new ideas for the security testing of complex stateful protocols and laying a solid foundation for improving the overall security and reliability of software systems.

[0054] 4. This invention effectively solves the problems of blind mutation and low efficiency in traditional fuzz testing. By intelligently evaluating candidate states and implementing targeted mutation tests based on UCB values, it significantly enhances the exploratory capabilities of fuzz testing and improves the effectiveness of test results. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a stateful protocol fuzz testing method based on reinforcement learning provided in an embodiment of the present invention.

[0056] Figure 2This is a schematic diagram illustrating an example of a Simple Mail Transfer Protocol (SMTP) message provided in an embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram illustrating the establishment and updating of a state model containing message information provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of a seed pool example provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the process for performing fuzz testing based on seed sequences provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the confidence interval provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the experience-based implementation of segmented mutation provided in an embodiment of the present invention;

[0062] Figure 8 This is the schematic diagram of the process for implementing segmented mutation based on Thompson sampling provided in the embodiments of the present invention. Detailed Implementation

[0063] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0064] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0065] Example 1

[0066] like Figure 1 As shown in the figure, this embodiment introduces a stateful protocol fuzz testing method based on reinforcement learning, including:

[0067] Step 1: Construct a state model containing message information.

[0068] Capture network traffic generated by the test target during communication;

[0069] The network traffic generated by the test target during communication is input into a state model containing message information. A seed sequence including state information and message content is extracted and the seed sequence is output.

[0070] The method for constructing the state model containing message information includes:

[0071] Capture the network traffic generated by the test target during each communication and extract data packets from the network traffic.

[0072] Based on the source IP address and destination IP address of the data packet, the data packet is divided into server response messages and client service request messages.

[0073] Keywords are extracted from the server's response message to represent the server's status, and an initial status model is established based on the keywords. The keywords include status codes and text content.

[0074] The server's response messages reflect the status of the service program, while the service request messages sent by the client are objects that need to be modified. Taking the Simple Mail Transfer Protocol (SMTP) as an example, the interaction messages between the client and the server are as follows: Figure 2 As shown.

[0075] Extract the application layer payload from the client service request message, perform mutation processing to obtain the mutated payload, encapsulate the mutated payload back into the client service request message data packet, and send the encapsulated client service request message data packet to the server for testing;

[0076] The client service request message and the initial state model are merged to obtain a state model containing message information.

[0077] A state model containing message information is used to extract seed sequences, which are then added to a seed pool. The seed sequence is the input to the fuzzing method of this invention, and it contains state information and related message content.

[0078] The state model containing message information provided by this invention is gradually updated and improved during fuzz testing. The initial state model and the state model containing message information are stored in a graph structure. Each node in the graph structure represents a state of the test target, and the edges between nodes represent messages that cause the test target state to change.

[0079] like Figure 3As shown, when the state of the test target changes to a new state, the new state is inserted into the graph structure as a new node. For example, when the new state S7 of the test target is discovered, sending message M' under the known state S2 can transition to the new state S7. By sending the message "QUIT", the server transitions from the new state S7 to the known state S6. The new state S7 is inserted into the graph as a new node, indicating that the predecessor of the node representing the known state S2 is the node with the associated message M'; the successor of the node representing the new state S7 is the known state S6, with the associated message QUIT.

[0080] The establishment of a state model containing message information is for the purpose of specifically obtaining seed sequences. The seed sequence provided by this invention is a sequence containing state information and related message information, which is extracted using a state model containing message information. The seed sequence is not a single message, but a sequence, which can be intuitively represented as a path in the structure diagram of the state model containing message information.

[0081] This invention uses a seed pool to store seed sequences, such as Figure 4 As shown, after extracting the seed sequence from the state model containing message information, it is added to the seed pool. The start and end states are selected according to the state machine. Generally, state nodes without predecessors and state nodes without successors are selected as the start and end states.

[0082] After selecting the start and end states, the structure graph representing the state model containing message information is traversed, and all paths from the start node to the end node are output as seed sequences and stored in the seed pool. As more nodes are added to the structure graph representing the state model containing message information, the seed pool is updated accordingly. Figure 3 For example, when the node representing the new state S7 and the node representing the new state S8 are added to the structure graph of the state model containing message information, there are three paths between the node representing the known state S1 and the node representing the known state S6, and the path information will be updated to the seed pool accordingly.

[0083] As fuzzing continues, new states of the test target and trigger messages are added to the state model containing message information for updating, and the message sequence that triggers the new state is added as a new seed sequence to the seed pool, making the state model containing message information richer and more complete.

[0084] Step 2: Mutate the seed sequence to generate test cases.

[0085] The method for generating test cases by mutating the seed sequence includes:

[0086] Step 2.1: Retrieve a seed sequence from the seed pool;

[0087] Step 2.2: For the seed sequence, input the seed sequence into the fuzzer, evaluate the candidate states of the test target, and select the target state to be tested from the candidate states. The specific steps are as follows:

[0088] The UCB algorithm is used to evaluate each candidate state in the seed sequence and calculate the UCB value of each candidate state.

[0089] The candidate state with the highest UCB value is taken as the target state for the current test.

[0090] In this embodiment, the test target is guided to the target state by sending a guiding message in the seed sequence, in preparation for fuzz testing of the test target.

[0091] This invention utilizes the UCB algorithm to select the target state based on a confidence interval, where the confidence interval is the range of reward values ​​for each candidate state in the seed sequence, and the reward value is estimated accordingly.

[0092] like Figure 6 As shown, a certain number of state explorations have been accumulated during the fuzzy testing process. The reward value table is updated, and the mean of the reward distribution is estimated based on historical reward data. The actual rewards are distributed around the mean. of Within the neighborhood, among which Indicates uncertainty factor, The upper confidence bound represents the maximum estimated value of the reward value of the corresponding candidate state. The upper confidence bound is used as the benchmark for selecting the candidate state as the target state, that is, the target state of the current test is selected based on the maximum estimated value.

[0093] The target state for the current test is determined based on the upper bound of the state confidence interval. Each time, the candidate state with the largest upper confidence bound is selected as the target state for testing. At the beginning of the test, all candidate states have the same upper confidence bound, and at this point, all candidate states have the same value as the target state. During the test, this embodiment adds an uncertainty factor to the mean of the historical upper confidence bounds of each state. Update the upper confidence bound.

[0094] If, when testing a target state, the target can enter a new state not included in the seed sequence, the confidence upper bound of the target state is positively increased. In this embodiment, the target state will be the primary target state to be tested.

[0095] In the fuzz testing process, this embodiment utilizes the "experience" accumulated from previous tests to perform multiple rounds of state selection for a seed sequence. Each round selects a target state for testing, and the next round's state selection utilizes the reward value information accumulated from the previous round. With continuous iterative testing, the uncertainty factor... As the growth rate decreases, even less frequently selected states in the seed sequence have the opportunity to be chosen as target states for testing.

[0096] Specifically, this invention uses the UCB algorithm to estimate the UCB value of each candidate state in the seed sequence, obtaining the UCB value of each candidate state. The formula for calculating the UCB value is expressed as:

[0097] (1);

[0098] In the formula, UCB i This represents the UCB value of the i-th candidate state. This indicates that the UCB value is positive infinity; "if the ithstate has never been selected" means the state of the test target has never been selected; t represents the round in which the UCB algorithm performs the estimation; and "otherwise" means "otherwise". Let represent the estimated mean of the i-th candidate state after being selected in round t-1, and let represent the "experience" accumulated by the i-th state through t-1 rounds of testing. This experience will serve as the basis for updating the UCB value. During testing, estimation refers to measuring the value of selecting the current state for testing based on historical experience, while selection refers to selecting the state with the highest estimated value for testing. Before round t of estimation, this embodiment has performed a total of ... Second choice This will affect the estimation in round t to some extent, where it is assumed that the i-th state is selected. This time, and the estimated mean of this state is If this state is never selected, then UCB is set to a larger value, as used in the formula. This indicates that... Otherwise, UCB i equal + .

[0099] (2);

[0100] In the formula, denoted by , where represents the number of times the i-th state of the test target is selected, log represents the logarithmic function, and n is the total number of estimations performed by the UCB algorithm.

[0101] In summary, during fuzz testing, the selection of the target state in this invention is mainly based on the following two considerations:

[0102] First, focus on identifying and prioritizing testing states deemed "interesting." "Interesting" states are those where testing can induce the tested system into a previously unknown state. Since vulnerabilities and protocol states are often closely correlated, many vulnerabilities are only triggered under specific system states. Therefore, in-depth testing of these "interesting" states increases the chances of covering more new states, thereby improving the likelihood of discovering potential vulnerabilities. When selecting target states, those that can guide the system into new states will be given priority.

[0103] Meanwhile, to ensure comprehensive testing, this invention does not overlook states that are less frequently selected in the seed sequence. During fuzz testing, in addition to pursuing in-depth exploration of the "interesting" states, it is also necessary to ensure sufficient coverage of all states. This is because even if some states seem less "interesting" in the current test, they may still hide undiscovered problems or vulnerabilities. A target state selection strategy ensures that less frequently selected states in the seed sequence also receive testing opportunities, further improving the comprehensiveness and effectiveness of the testing.

[0104] Step 2.3: Based on the Thompson sampling algorithm, perform segmented mutation on the seed sequence of the selected state to obtain multiple segments.

[0105] like Figure 8 As shown in this embodiment, before performing segmentation mutation on the message corresponding to the selected target state, it is necessary to first segment the message corresponding to the selected target state to obtain multiple segments, including:

[0106] If the message contains special symbols, the message is divided into multiple segments based on the special symbols.

[0107] When the message does not contain special symbols, the message is divided into multiple segments using a fixed-length segmentation method.

[0108] For example, it can be divided into 4-bit segments, and the specific length can be adjusted according to actual needs.

[0109] Based on the Thompson sampling algorithm, a segment is selected from the multiple segments to perform segment mutation, resulting in a mutated segment and an unmutated segment. The mutated segment and the unmutated segment are then concatenated to generate test cases.

[0110] Send the test cases to the test target for testing, and repeat the following steps until the preset test termination condition is met:

[0111] When the state of the test target changes, the reward mechanism is triggered, and the reward value is accumulated.

[0112] The sampling probability of the segment that mutated in the current step is updated based on the cumulative reward value using a fuzzer.

[0113] Select the segment with the highest sampling probability for mutation, and then combine the mutated segment and the unmutated segment in the current step to construct a new test case.

[0114] Send new test cases to the test target for testing and monitor changes in the test target's status.

[0115] Specifically, when the test target changes to a new state, the new state of the test target and the trigger message are added to the state model containing message information for updating, and the message sequence that triggers the new state is added to the seed pool as a new seed sequence.

[0116] When the state of the test target changes, a reward mechanism is triggered, accumulating reward values. A fuzzer is then used to update the sampling probability of the mutated segment in the current step based on the accumulated reward values. The rules of the reward mechanism are as follows:

[0117] When the test target changes to a new state, the reward value of the segment that mutated in the current step is increased by "1";

[0118] When the test target changes to a known state, the reward value of the segment that mutated in the current step is incremented by "0".

[0119] The reward mechanism rule is expressed as:

[0120] (3);

[0121] In the formula, reward i+1 Represents the reward value for the (i+1)th test. i This represents the reward value for the i-th test, if means "if", new_state means the new state, True means a new state was found, and False means no new state was found.

[0122] To eliminate interference from certain special states, the reward mechanism designed in this embodiment also includes the following rules:

[0123] When the test target enters an abnormal handling state, the reward value of the segment that mutated in the current step is incremented by "0";

[0124] When communicating via a client / server architecture protocol, the reward value of the segment that has mutated in the current step is dynamically updated based on the server's real-time response to the client's request.

[0125] For publicly available protocols where the status specification is public and exception handling has clear identifying characteristics, such as in the FTP protocol where 501 indicates that the service program cannot recognize the request method, this type of response code is usually due to invalid test cases.

[0126] For example, if the mutation of the "USER anonymous" message focuses on the "USER" segment, the resulting test case might be similar to "ü\u0098\u009B\v\020òØ\000\004¸\v\000 anonymous\r\n". Sending such a test case will result in a "Syntax error" response. Mutations on the "USER" segment are unlikely to generate test cases that can be accepted by the server program; they only keep the program in an exception handling state, which is not conducive to generating effective test cases or exploring new states of the test target. Therefore, when such a state is matched, this embodiment adjusts the reward value to "0".

[0127] Protocol communication programs are generally based on a client / server architecture, and responses are received after a request is sent. The value of each segment can be updated in real time based on the response, and the segment mutation strategy is dynamically adjusted based on the score of each segment.

[0128] The method for accumulating the reward value includes:

[0129] Repeat the following steps until the number of tests for the message corresponding to the currently selected target state reaches the test termination condition:

[0130] Record the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0";

[0131] Based on the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0", the sampling probability of the segment in the message that has mutated is calculated using the β distribution probability density function.

[0132] In summary, this embodiment treats the fuzzing process as a multi-round trial. For each seed sequence, multiple rounds of state selection are performed, with each round selecting a target state for testing. The next round's state selection is based on the "experience" accumulated from previous tests. During iterative testing, the fuzzer can autonomously accumulate experience and adjust its strategy based on the feedback obtained from the fuzzing process. In this embodiment, the fuzzer has already explored N-1 times before the Nth test begins.

[0133] like Figure 7As shown, before each round of attempts, the blurr analyzes the data based on its accumulated experience to determine which parts of the packets are more valuable for mutation, and prioritizes the mutation of those parts. By leveraging its accumulated experience, the blurr can perform tests more effectively, increasing the proportion of valid test cases while reducing the generation of redundant test cases, thus improving its ability to discover security vulnerabilities.

[0134] Based on segmented mutation, the Thompson sampling algorithm can calculate the sampling probability of each segment by using the β distribution probability density function based on the cumulative number of times the reward value is increased by "1" and "0" during the fuzzy testing process, and select the best segment for mutation.

[0135] Each seed sequence has a threshold for the number of test rounds. Once the threshold is reached, the next seed sequence will be selected for testing.

[0136] The flowchart of fuzz testing based on seed sequences provided in this embodiment of the invention is as follows: Figure 5 As shown, the messages in the seed sequence are divided into three categories: boot messages, target messages, and remaining messages.

[0137] The seed sequence is expressed as formula (1):

[0138] (1);

[0139] In the formula, Seed represents the complete seed sequence, msg1 represents the bootstrap message, msg2 represents the target message, and msg3 represents the remaining messages.

[0140] The mutated seed sequence is represented by formula (2):

[0141] (2);

[0142] In the formula, Seed mutated This represents the mutated seed sequence, and mutated(msg2) represents the mutated target message.

[0143] As shown in formula (1), msg1 is the guiding message used to guide the seed sequence to the target state. As shown in formula (2), after guiding the seed sequence to the target state, the target message msg2 is mutated, and the mutated target message mutated(msg2) is sent to the target program. After the mutated message is sent, in order to make full use of the captured messages, the remaining message msg3 is also sent to the target program.

[0144] For example, in a test of an FTP server, the boot message msg1 is "USER test", the target message msg2 is "PASS testpass", and the remaining messages msg3 include "XMKD", "NLIST", and "QUIT". Boot message msg1 guides the service program to "331 Please specify the password". The test program mutates the target message msg2 "PASS testpass". After sending the mutated message (msg2), the server enters the "530 Please login with USER and PASS" state. Then, the remaining messages msg3 are sent. Sending "XMKD" and "NLIST" does not cause a state change. After sending "QUIT", the server responds with "221 Goodbye." Finally, a reset message is sent to reset the service program for the next round of testing.

[0145] In summary, the method of this invention employs Thompson sampling as the baseline strategy for segmented mutation. Thompson sampling, based on Bayesian principles, offers strong real-time performance. The prior distribution is primarily derived from experience accumulated in historical fuzzy testing, and the posterior distribution is updated based on the prior distribution information for each segment. Sampling is performed based on the posterior distribution to obtain a value that conforms to that distribution, i.e., the sampled value. The segment to be mutated is selected based on the sampled value.

[0146] The distribution-based selection method described above exhibits strong adaptability. When selecting segments, it can achieve a balance between segments that have already yielded high rewards and segments that have not yet been selected. Intuitively, when selecting certain segments yields higher rewards, the posterior distribution of these segments will change positively, and the probability of them being selected will increase.

[0147] In the above steps, when a new state of the test target is triggered, the new state of the test target and the trigger message are added to the state model, and the message sequence that triggers the new state is added to the seed pool as a new seed sequence.

[0148] Step 3: Use the test cases generated by message mutation to test the target.

[0149] Example 2

[0150] A computer-readable storage medium storing computer instructions thereon, characterized in that the computer instructions, when executed by a processor, implement the steps of the method described in Embodiment 1 above.

[0151] Example 3

[0152] The present invention also provides a computer program product, including computer instructions, characterized in that the computer instructions, when executed by a processor, implement the steps of the method described in Embodiment 1 above.

[0153] In summary, this implementation involves two core strategies: a sampling strategy and a reward update strategy. The sampling strategy and the reward update strategy are interconnected and work together.

[0154] The sampling strategy is primarily used to guide segment selection, that is, selecting segments that need to be mutated based on currently accumulated experience. Segment selection is implemented based on the β distribution probability density function, which characterizes the probability of a segment being selected. The β distribution probability density function is suitable for situations where parameters are uncertain. In this embodiment, segment selection is treated as a selection problem based on a probability distribution. Each segment corresponds to a parameter, which is sampled based on the β distribution and used as the basis for segment selection. The larger the parameter value, the higher the probability of the segment being selected. As fuzzing progresses, test cases are constructed by mutating the selected segments. After being sent to the test target, reward values ​​are collected and fed back to the fuzzer. The fuzzer will use the reward values ​​as experience to synchronously update the β distribution and continuously use the β distribution to guide segment selection.

[0155] The core of the reward update strategy lies in adjusting the rewards for each segment based on changes in the test target's state. After selecting the segment to be mutated, the mutated data replaces the data in the selected segment to construct a new test case. This new test case is then sent to the test target, and the reward value of the mutated segment in the current step is updated based on the test target's response. Based on the state coverage principle, if a test case triggers a new state, the reward value of the mutated segment in the current step is incremented by "1"; otherwise, it is incremented by "0". The aim is to explore new states by continuously mutating the segment data in the message and dynamically adjust the reward value of each segment in the message based on whether the mutation of a particular segment can trigger a new state of the test target, thereby improving testing efficiency and effectiveness.

[0156] This invention aims to improve protocol fuzzing efficiency by optimizing state selection and message mutation during the fuzzing process through an innovative protocol state coverage-oriented fuzzing method. The core of this protocol state coverage-oriented method lies in its ability to intelligently focus fuzzing on protocol states and message segments that are likely to reveal new states of the test target during state selection and message segment mutation. This invention not only effectively reduces the number of invalid tests and avoids wasting resources, but also significantly improves overall testing efficiency, ensuring that the testing process is more focused on critical protocol states and message segments.

[0157] This invention proposes an innovative reinforcement learning-based fuzzing method for stateful protocols, deeply integrating reinforcement learning theory with fuzzing techniques. By introducing a reinforcement learning mechanism and combining the state selection strategy of the UCB algorithm with the segmented mutation technique of the Thompson sampling algorithm, the intelligence and efficiency of stateful protocol fuzzing are significantly improved. The method provided by this invention not only guides the test to cover new states of the test target but also provides feedback on the test results through a reward mechanism, achieving precise mutation and optimization of test cases. The testing process of this invention significantly reduces the generation of redundant test cases, making the fuzzing process more efficient and targeted. Simultaneously, the dynamic updating of the state model and the expansion of the seed pool ensure comprehensive test coverage, effectively improving the depth and breadth of fuzzing and providing strong technical support for vulnerability discovery.

[0158] This invention effectively solves the problems of blind mutation and low efficiency in traditional fuzz testing. By intelligently evaluating candidate states and implementing targeted mutation operations based on UCB values, it significantly enhances the exploratory capabilities of fuzz testing and improves its effectiveness. Simultaneously, by dynamically adjusting the testing strategy according to the state transitions of the test target, it achieves continuous optimization of the testing process, providing a new approach to security testing of complex stateful protocols and laying a solid foundation for improving the overall security and reliability of software systems.

[0159] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A stateful protocol fuzzy testing method based on reinforcement learning, characterized in that, include: Capture network traffic generated by the test target during communication; The network traffic generated by the test target during communication is input into a state model containing message information. A seed sequence including state information and message content is extracted and the seed sequence is output. The seed sequence is mutated to generate test cases for testing the target. The method for generating test cases by mutating the seed sequence includes: The seed sequence is input into the fuzzer to evaluate the candidate states of the test target and select the target state to be tested from the candidate states; By sending the pilot message in the seed sequence, the test target is guided to the target state; The message corresponding to the selected target state is divided into multiple segments according to special symbols; Based on the Thompson sampling algorithm, a segment is selected from the multiple segments to perform segment mutation, resulting in a mutated segment and an unmutated segment. The mutated segment and the unmutated segment are then concatenated to generate test cases. Send the test cases to the test target for testing, and repeat the following steps until the preset test termination condition is met: When the state of the test target changes, the reward mechanism is triggered, and the reward value is accumulated. The sampling probability of the segment that mutated in the current step is updated based on the cumulative reward value using a fuzzer. Select the segment with the highest sampling probability for mutation, and then combine the mutated segment and the unmutated segment in the current step to construct a new test case. Send new test cases to the test target for testing and monitor changes in the test target's status. Specifically, when the state of the test target changes to a new state, the new state of the test target and the trigger message are added to the state model containing message information for updating, and the message sequence that triggers the new state is added to the seed pool as a new seed sequence.

2. The stateful protocol fuzzy testing method based on reinforcement learning according to claim 1, characterized in that, The method for constructing the state model containing message information includes: Capture the network traffic generated by the test target during each communication, and extract data packets from the network traffic; Based on the source IP address and destination IP address of the data packet, the data packet is divided into server response messages and client service request messages; Extract keywords from the server response message to represent the server's status and establish an initial status model based on the keywords. The keywords include status codes and text content. Extract the application layer payload from the client service request message, perform mutation processing to obtain the mutated payload, encapsulate the mutated payload back into the client service request message data packet, and send the encapsulated client service request message data packet to the server for testing; The client service request message and the initial state model are merged to obtain a state model containing message information.

3. The stateful protocol fuzzy testing method based on reinforcement learning according to claim 2, characterized in that, The initial state model and the state model containing message information are stored in a graph structure. Each node in the graph structure represents a state of the test target, and the edges between nodes represent messages that cause the test target state to change. When a new state of the test target appears, the new state is inserted into the graph structure as a new node.

4. The stateful protocol fuzzy testing method based on reinforcement learning according to claim 1, characterized in that, The method for selecting the target state includes: The UCB algorithm is used to evaluate the candidate states of the test target and calculate the UCB value of each candidate state. The candidate state with the highest UCB value is taken as the target state for the current test.

5. The stateful protocol fuzz testing method based on reinforcement learning according to claim 1, characterized in that, Methods for segmenting the message corresponding to the selected target state into multiple segments according to special symbols include: If the message contains special symbols, the message is divided into multiple segments based on the special symbols. When the message does not contain special symbols, the message is divided into multiple segments using a fixed-length segmentation method.

6. The stateful protocol fuzzy testing method based on reinforcement learning according to claim 1, characterized in that, When the state of the test target changes, a reward mechanism is triggered, accumulating reward values. A fuzzer is then used to update the sampling probability of the mutated segment in the current step based on the accumulated reward values. The rules of the reward mechanism are as follows: When the test target changes to a new state, the reward value of the segment that has mutated is incremented by "1". When the test target changes to a known state, the reward value of the segment that has mutated is incremented by "0".

7. The stateful protocol fuzz testing method based on reinforcement learning according to claim 6, characterized in that, The rules of the reward mechanism also include: When the test target enters an abnormal handling state, the reward value of the segment that is currently mutated is incremented by "0"; When communicating using a client / server architecture protocol, the reward value of the currently mutated segment in the message is dynamically updated based on the server's real-time response to the client's request.

8. The stateful protocol fuzz testing method based on reinforcement learning according to claim 7, characterized in that, The method for accumulating the reward value includes: Repeat the following steps until the number of tests for the message corresponding to the currently selected target state reaches the test termination condition: Record the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0"; Based on the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0", the sampling probability of each segment is calculated using the β distribution probability density function.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-8.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-8.

Images