State protocol fuzzy testing method based on reinforcement learning, medium and product

Abstract

The invention discloses a stateful protocol fuzzy test method based on reinforcement learning, a medium and a product, and the method comprises the steps: inputting the network flow of a test target into a state model containing message information, extracting seed sequence variation, and generating a test case, including evaluating a candidate state of the test target and selecting a target state; obtaining a message sequence guided to a target state, and implementing segmentation variation to obtain a plurality of segments; the fuzzer synchronously updates the sampling probability of the segments according to the accumulation times of the reward values, and selects the segments with the sampling probability exceeding a threshold value to generate a test case; and when the new state of the test target is triggered in the test process, adding the new state and the trigger message into the state model containing the message information, and adding the message sequence which triggers the new state as a new seed into the seed pool for updating. According to the invention, the pertinence and the efficiency of fuzzy testing are effectively improved.

Description

Technical Field

[0001] The present invention relates to a stateful protocol fuzz testing method, medium and product based on reinforcement learning, belonging to the technical field of network protocol fuzz testing. Background Art

[0002] Fuzz testing technology is widely used in network protocol vulnerability mining. The core of fuzz testing of network protocols is to automatically or semi-automatically construct abnormal message data and send it to the network communication program, and monitor the working status of the tested program in real time (crash, service interruption, etc.) to discover security vulnerabilities in the network communication program. According to the different test case construction methods, protocol fuzz testing technology can be divided into two categories: generation-based fuzz testing and mutation-based fuzz testing.

[0003] In the generation-based fuzz testing technology, the construction of test cases depends on the protocol description model provided by the tester. Before implementing fuzz testing, the tester needs to spend a lot of effort to obtain information such as the protocol format and protocol state machine. Although the existence of automated protocol reverse engineering tools reduces the difficulty of protocol analysis, the accuracy of the analysis is difficult to meet the needs of fuzz testing.

[0004] The mutation-based fuzz testing method mutates the captured network messages to construct test cases, without the need for the tester to analyze the protocol. When performing fuzz testing on stateful protocols, the mutation-based fuzz testing method often guides the protocol program to a specific state through the guide message, and then mutates the subsequent messages to carry out the test.

[0005] Depending on the information required for fuzz testing, fuzz testing can be divided into gray-box fuzz testing and black-box fuzz testing. Existing gray-box fuzz testing methods often use the method of instrumenting the source code to obtain the internal information of the target program, and track memory allocation and network I / O flow through stub feedback. This type of method can accurately understand the protocol status by obtaining internal feedback during program execution, but it also has some limitations. First, existing methods mostly use the method of instrumenting at compile time to instrument the program. If the source code of the target program cannot be obtained, this method will be difficult to use. Secondly, the instrumentation method needs to maintain probe data in memory, and the recovery of the state will also consume resources, which is difficult to apply under conditions of limited resources. In contrast, black-box fuzz testing does not rely on source code. The tester only needs to send test cases to the program under test and monitor the performance of the program under test after processing the test cases. It has a wider range of applications.

[0006] The existing mutation-based protocol black-box fuzz testing methods mainly have the following problems: First, the mutation of the message generally adopts sequential or random operations. Such operations are blind and cannot focus the mutation on certain key fields to ensure that the use cases generated by the mutation explore a wider state space. Secondly, the state selection is blind. Existing methods often use random methods or rule-based state selection. Using random strategies, it is difficult to focus fuzz testing on states with in-depth testing value. Testing with fixed rules can easily cause state selection to fall into local optimality. In addition, most fuzz testing methods do not make full use of the experience accumulated in the fuzz testing process to improve the efficiency of fuzz testing. These factors restrict the overall performance of fuzz testing.

[0007] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgement or any form of suggestion that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention

[0008] The technical problem to be solved by the present invention is: how to improve the pertinence and efficiency of fuzzy testing.

[0009] In order to achieve the above objectives / solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0010] In a first aspect, the present invention provides a stateful protocol fuzz testing method based on reinforcement learning, comprising:

[0011] Capture the network traffic generated by the test target during communication;

[0012] Input the network traffic generated by the test target during the communication process into a state model containing message information, extract a seed sequence including the state information and message content, and output the seed sequence;

[0013] Mutating the seed sequence to generate a test case for testing the test target;

[0014] The method of mutating the seed sequence to generate a test case includes:

[0015] Input the seed sequence into the fuzzifier, evaluate candidate states of the test target and select a target state for the current test from the candidate states;

[0016] By sending the guide message in the seed sequence, the test target is guided to the target state;

[0017] The message corresponding to the selected target state is divided into multiple segments according to special symbols;

[0018] Based on the Thompson sampling algorithm, a segment is selected from the multiple segments to perform segment mutation, so as to obtain a mutated segment and an unmutated segment, and the mutated segment and the unmutated segment are spliced ​​to generate a test case;

[0019] Send the test case to the test target for testing, and repeat the following steps until the preset test termination condition is reached:

[0020] When the state of the test target changes, the reward mechanism is triggered and the reward value is accumulated;

[0021] Use the fuzzifier to update the sampling probability of the segment that has mutated in the current step according to the accumulation of reward values;

[0022] Select a segment with the highest sampling probability for mutation, and concatenate the mutated segment and the unmutated segment in the current step to construct a new test case;

[0023] Send new test cases to the test target for testing and monitor the status changes of the test target;

[0024] When the state of the triggering test target changes to a new state, the new state of the test target and the triggering message are added to the state model containing message information for updating, and the message sequence triggering the new state is added to the seed pool as a new seed sequence.

[0025] Furthermore, the method for constructing the state model including message information includes:

[0026] Capturing network traffic generated by the test target during each communication process, and extracting data packets from the network traffic;

[0027] According to the source IP address and the destination IP address of the data packet, the data packet is divided into a server response message and a client service request message;

[0028] Extracting keywords from the server response message to represent the state of the server and establishing an initial state model based on the keywords, wherein the keywords include a state code and text content;

[0029] Extract the application layer payload of the client service request message, perform mutation processing to obtain the mutated payload, encapsulate the mutated payload back into the client service request message data packet, and send the encapsulated client service request message data packet to the server for testing;

[0030] The client service request message and the initial state model are merged to obtain a state model containing message information.

[0031] Furthermore, the initial state model and the state model containing message information are stored in a graph structure, each node in the graph structure represents a state of the test target, and the edges between the nodes represent the messages that cause the test target state to change. When a new state appears in the test target, the new state is inserted into the graph structure as a new node.

[0032] Furthermore, the method for selecting the target state includes:

[0033] Evaluate the candidate states of the test target based on the UCB algorithm and calculate the UCB value of each candidate state;

[0034] The candidate state with the highest UCB value is taken as the target state for the current test.

[0035] Furthermore, the method of dividing the message corresponding to the selected target state into multiple segments according to special symbols includes:

[0036] When there are special symbols in the message, the message is divided according to the special symbols in the message to obtain multiple segments;

[0037] When there is no special symbol in the message, the message is divided into multiple segments by using a fixed-length division method.

[0038] Furthermore, the rules of the reward mechanism are:

[0039] When the trigger test target changes to a new state, the reward value of the segment that mutates in the current step is increased by "1";

[0040] When the trigger test target changes to a known state, the reward value of the segment that mutates in the current step is added with "0".

[0041] Furthermore, the rules of the reward mechanism also include:

[0042] When the test target enters the exception handling state, the reward value of the segment that has mutated in the current step is added with "0";

[0043] When the protocol communicates based on the C / S architecture, the reward value of the segment that has mutated in the current step is dynamically updated according to the real-time response of the server to the client request.

[0044] Furthermore, the method for accumulating the reward value includes:

[0045] Repeat the following steps until the number of test times for the message corresponding to the currently selected target state reaches the test termination condition:

[0046] Record the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0";

[0047] According to the cumulative number of times the reward value is added with "1" and the reward value is added with "0", the sampling probability of the segment in which the mutation occurs in the message is calculated using the beta distribution probability density function.

[0048] In a second aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of the method described in the first aspect.

[0049] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. This paper proposes an innovative stateful protocol fuzz testing method based on reinforcement learning, which deeply integrates reinforcement learning theory and fuzz testing technology. By introducing the reinforcement learning mechanism, combining the state selection strategy with the segmented mutation technology of the Thompson sampling algorithm, the intelligence and efficiency of stateful protocol fuzz testing are significantly improved.

[0052] 2. The method provided by the present invention can not only guide the test to cover new states and known states that have not been tested, but also feedback the test results through the reward mechanism, thus realizing the precise variation and optimization of test cases. The present invention greatly reduces the generation of redundant test cases, making the fuzzy testing process more efficient and targeted.

[0053] 3. At the same time, the present invention can dynamically update the state model and expand the seed pool, ensuring comprehensive test coverage, effectively improving the depth and breadth of vulnerability mining, and providing strong technical support for vulnerability mining. At the same time, according to the state transition of the test target, the test strategy is dynamically adjusted to achieve continuous optimization of the test process, providing new ideas for the security testing of complex stateful protocols, and laying a solid foundation for improving the overall security and reliability of the software system.

[0054] 4. The present invention effectively solves the problem of blind mutation and low efficiency in traditional fuzz testing. By intelligently evaluating candidate states and implementing targeted mutation testing based on UCB values, the exploration capability of fuzz testing is significantly enhanced, and the effectiveness of test results is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of a stateful protocol fuzz testing method based on reinforcement learning provided by an embodiment of the present invention.

[0056] Figure 2This is a schematic diagram of an example of a Simple Mail Transfer Protocol (SMTP) message provided by an embodiment of the present invention;

[0057] Figure 3 It is a schematic diagram of establishing and updating a state model including message information provided by an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of an example of a seed pool provided by an embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of a process of performing fuzzy testing based on a seed sequence provided by an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of a confidence interval provided by an embodiment of the present invention;

[0061] Figure 7 It is a schematic diagram of a process of implementing segmented variation based on experience provided by an embodiment of the present invention;

[0062] Figure 8 This is the process intention of implementing segmented variation based on Thompson sampling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0064] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0065] Example 1

[0066] like Figure 1 As shown, this embodiment introduces a stateful protocol fuzz testing method based on reinforcement learning, including:

[0067] Step 1: Build a state model containing message information.

[0068] Capture the network traffic generated by the test target during communication;

[0069] The network traffic generated by the test target during the communication process is input into a state model containing message information, a seed sequence including the state information and message content is extracted, and the seed sequence is output.

[0070] The method for constructing the state model containing message information comprises:

[0071] Capture the network traffic generated by the test target during each communication process and extract data packets from the network traffic.

[0072] According to the source IP address and the destination IP address of the data packet, the data packet is divided into a server response message and a client service request message.

[0073] Keywords are extracted from the server response message to represent the state of the server and an initial state model is established based on the keywords, wherein the keywords include a state code and text content.

[0074] The response message of the server can reflect the status of the service program. The service request message sent by the client is the object that needs to be mutated. Taking the simple mail transfer protocol SMTP as an example, the interactive message between the client and the server is as follows: Figure 2 shown.

[0075] Extract the application layer payload of the client service request message, perform mutation processing to obtain the mutated payload, encapsulate the mutated payload back into the client service request message data packet, and send the encapsulated client service request message data packet to the server for testing;

[0076] The client service request message and the initial state model are merged to obtain a state model containing message information.

[0077] The state model containing the message information is used to extract the seed sequence, and the seed sequence will be added to the seed pool. The seed sequence is the input of the fuzzy testing method of the present invention, and the seed sequence contains the state information and the relevant message content.

[0078] The state model containing message information provided by the present invention is gradually updated and improved during the fuzzy testing process. The initial state model and the state model containing message information are stored in a graph structure. Each node in the graph structure represents a state of the test target, and the edges between the nodes represent the messages that prompt the test target state to change.

[0079] like Figure 3As shown, when the state of the test target changes to a new state, the new state is inserted into the graph structure as a new node. For example, when the new state S7 of the test target is explored, sending a message M' in the known state S2 can be converted to the new state S7. By sending the message "QUIT", the server side converts from the new state S7 to the known state S6, and the new state S7 is inserted into the graph as a new node. The predecessor of the node representing the new state S7 is the node representing the known state S2, and the edge is the associated message M'; the successor of the node representing the new state S7 is the known state S6, and the edge is the associated message QUIT.

[0080] The establishment of the state model containing message information is to obtain the seed sequence in a targeted manner. The seed sequence provided by the present invention is a sequence containing state information and related message information, which is extracted using the state model containing message information. The seed sequence is not a separate message, but a sequence, which can be intuitively represented as a path in the structure diagram of the state model containing message information.

[0081] The present invention uses a seed pool to store seed sequences, such as Figure 4 As shown, a seed sequence is extracted from a state model containing message information and added to a seed pool. A start state and an end state are selected according to a state machine. Generally, a state node without a predecessor and a state node without a successor are selected as the start state and the end state.

[0082] After selecting the start state and the end state, the structure graph representing the state model containing the message information is traversed, and all paths from the start node to the end node are output as seed sequences and stored in the seed pool. As the number of nodes in the structure graph representing the state model containing the message information increases, the seed pool is also updated accordingly. Figure 3 For example, when the node representing the new state S7 and the node representing the new state S8 are added to the structural diagram of the state model containing message information, three paths are included between the node representing the known state S1 and the node representing the known state S6, and the path information will be updated to the seed pool accordingly.

[0083] As fuzz testing continues, the new state and trigger message of the test target are added to the state model containing message information for update, and the message sequence that triggers the new state is added to the seed pool as a new seed sequence, making the state model containing message information richer and more complete.

[0084] Step 2: Mutate the seed sequence to generate test cases.

[0085] The method for generating test cases by mutating the seed sequence includes:

[0086] Step 2.1: Take a seed sequence from the seed pool;

[0087] Step 2.2: For the seed sequence, input the seed sequence into the fuzzifier, evaluate the candidate states of the test target and select the target state of the current test from the candidate states. The specific steps are:

[0088] Evaluate each candidate state in the seed sequence based on the UCB algorithm and calculate the UCB value of each candidate state;

[0089] The candidate state with the highest UCB value is taken as the target state for the current test.

[0090] Among them, this embodiment guides the test target to the target state by sending the guide message in the seed sequence, and prepares to implement fuzzy testing on the test target.

[0091] The present invention uses the UCB algorithm to select the target state based on the confidence interval, where the confidence interval is the value range of the reward value of each candidate state in the seed sequence, and the estimation is performed based on the reward value.

[0092] like Figure 6 As shown in Figure 1, a certain number of state explorations have been accumulated during the fuzzy testing process, and the reward value table is updated. The mean of the reward distribution is estimated based on the historical data of the reward. The actual reward distribution falls within the mean of In the neighborhood, represents the uncertainty factor, The confidence upper bound represents the maximum estimated value of the reward value of the corresponding candidate state. The confidence upper bound is used as the basis for the candidate state to be selected as the target state, that is, the target state of the current test is selected based on the maximum estimated value.

[0093] The target state of the current test is determined based on the upper bound of the state confidence interval. Each time, the candidate state with the largest confidence upper bound is selected as the target state for testing. At the beginning of the test, all candidate states have the same confidence upper bound, and at this time, all candidate states have the same value as the target state. During the test, this embodiment adds the uncertainty factor based on the mean of the historical confidence upper bound of each state. Update the confidence upper bound.

[0094] If the test target can enter a new state not included in the seed sequence when testing in a certain target state of the test target, the confidence upper bound of the target state is increased positively, and this embodiment will focus on selecting the target state for testing.

[0095] During the fuzzy testing process, this embodiment uses the "experience" accumulated from previous tests to perform multiple rounds of state selection for a seed sequence. Each round of selection will select a target state for testing, and the next round of state selection will use the reward value information accumulated from the previous round of testing. With continuous iterative testing, the uncertainty factor The growth rate of is decreasing, so that the less selected states in the seed sequence also have the opportunity to be selected as the target state for testing.

[0096] Specifically, the present invention uses the UCB algorithm to estimate the UCB value of each candidate state in the seed sequence to obtain the UCB value of each candidate state, and the calculation formula of the UCB value is expressed as:

[0097] (1);

[0098] In the formula, UCB i represents the UCB value of the i-th candidate state, Indicates that the UCB value is positive infinity, if the ithstate has never been selected indicates that the state of the test target has never been selected, t indicates the round of estimation by the UCB algorithm, otherwise indicates "otherwise", It represents the estimated mean value of the i-th candidate state after being selected in the t-1th round, indicating that the i-th state has accumulated t-1 "experiences" in the t-1th selection and testing, and these experiences will serve as the basis for updating the UCB value. In the testing process, estimation refers to measuring the value of the current state selected for testing based on historical accumulated experience, and selection refers to selecting the state with the largest estimated value for testing. Before the tth round of estimation, this embodiment performs a total of Second choice, will affect the t-th round estimate to some extent, where, assuming that the i-th state is selected times, and the estimated mean of this state is , if this state has never been selected, then UCB is set to a larger value, and the formula uses Otherwise, UCB i equal + .

[0099] (2);

[0100] In the formula, represents the number of times the i-th state of the test target is selected, log represents the logarithmic function, and n is the total number of estimates performed by the UCB algorithm.

[0101] In summary, during the fuzzy testing process, the selection of the target state in the present invention is mainly based on the following two considerations:

[0102] First, focus on identifying and prioritizing testing of states that are considered "interesting". The "interesting" states refer to states where testing can cause the object under test to enter a previously unknown state. Since there is often a close correlation between vulnerabilities and protocol states, many vulnerabilities are only triggered in specific system states. Therefore, by deeply testing the "interesting" states, the chances of covering more new states can be increased, thereby increasing the possibility of discovering potential vulnerabilities. When selecting target states, states that can guide the system into a new state will be given priority.

[0103] At the same time, in order to ensure the comprehensiveness of the test, the present invention does not ignore the states that are less selected in the seed sequence. During fuzzy testing, in addition to pursuing in-depth exploration of the "interesting" states, it is also necessary to ensure sufficient coverage of all states. Because even if some states do not seem so "interesting" in the current test, they may still hide undiscovered problems or vulnerabilities. The target state selection strategy can ensure that the states that are less selected in the seed sequence also get a chance to be tested, which can further improve the comprehensiveness and effectiveness of the test.

[0104] Step 2.3: Based on the Thompson sampling algorithm, segmented mutation is performed on the seed sequence of the selected state to obtain multiple segments.

[0105] like Figure 8 As shown, in this embodiment, before implementing segmentation mutation on the message corresponding to the selected target state, it is necessary to first segment the message corresponding to the selected target state to obtain multiple segments, including:

[0106] When there are special symbols in the message, the message is divided according to the special symbols in the message to obtain multiple segments;

[0107] When there is no special symbol in the message, the message is divided into multiple segments by using a fixed-length division method.

[0108] For example, if the data is divided into 4 bits, the specific length can be adjusted according to actual needs.

[0109] Based on the Thompson sampling algorithm, a segment is selected from the multiple segments to perform segment mutation, so as to obtain a mutated segment and an unmutated segment, and the mutated segment and the unmutated segment are spliced ​​to generate a test case;

[0110] Send the test case to the test target for testing, and repeat the following steps until the preset test termination condition is reached:

[0111] When the state of the test target changes, the reward mechanism is triggered and the reward value is accumulated;

[0112] Use the fuzzifier to update the sampling probability of the segment that has mutated in the current step according to the accumulation of reward values;

[0113] Select a segment with the highest sampling probability for mutation, and concatenate the mutated segment and the unmutated segment in the current step to construct a new test case;

[0114] Send new test cases to the test target for testing and monitor the status changes of the test target;

[0115] When the trigger test target changes to a new state, the new state of the test target and the trigger message are added to the state model containing message information for updating, and the message sequence triggering the new state is added to the seed pool as a new seed sequence.

[0116] When the state of the test target changes, the reward mechanism is triggered, the reward value is accumulated, and the fuzzifier is used to update the sampling probability of the segment that has mutated in the current step according to the accumulation of the reward value; wherein the rule of the reward mechanism is:

[0117] When the trigger test target changes to a new state, add "1" to the reward value of the segment that has mutated in the current step;

[0118] When the trigger test target changes to a known state, add “0” to the reward value of the segment that mutated in the current step.

[0119] The expression of the reward mechanism rule is expressed as:

[0120] (3);

[0121] In the formula, reward i+1 Represents the reward value of the i+1th test, reward i Represents the reward value of the i-th test, if means "if", new_state represents the new state, True means that a new state is found, and False means that no new state is found.

[0122] In order to eliminate interference from some special states, the rules of the reward mechanism designed in this embodiment also include:

[0123] When the test target enters the exception handling state, add "0" to the reward value of the segment that has mutated in the current step;

[0124] When the protocol communicates based on the C / S architecture, the reward value of the segment that has mutated in the current step is dynamically updated according to the real-time response of the server to the client request.

[0125] For public protocols where the status specification is public and exception handling has obvious characteristic identification. For example, in the FTP protocol, 501 means that the service program cannot recognize the request method. Such response codes are usually caused by invalid test cases.

[0126] For example, if the mutation of the "USER anonymous" message focuses on the "USER" segment, a test case similar to "ü\u0098\u009B\v\020òØ\000\004¸\v\000 anonymous\r\n" may be generated after the mutation. When the test case is sent, a response "Syntax error" will be received. The mutation of the "USER" segment is difficult to generate a test case received by the server program, and the program can only continue to enter the exception handling state, which is not conducive to the generation of valid test cases and the exploration of new states of the test target. Therefore, when such a state is matched, this embodiment adjusts the reward value to "0".

[0127] The protocol communication program is generally based on a C / S architecture, and the response will be received after the request is sent. The value of each segment can be updated in real time based on the response, and the segment mutation strategy is dynamically adjusted based on the score of each segment.

[0128] The method for accumulating the reward value includes:

[0129] Repeat the following steps until the number of test times for the message corresponding to the currently selected target state reaches the test termination condition:

[0130] Record the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0";

[0131] According to the cumulative number of times the reward value is added with "1" and the reward value is added with "0", the sampling probability of the segment in which the mutation occurs in the message is calculated using the beta distribution probability density function.

[0132] In summary, this embodiment regards the fuzz testing process as a process of multiple rounds of attempts. Multiple rounds of state selection will be performed for each seed sequence. Each round of selection will select a target state for testing, and the next round of state selection will be based on the "experience" accumulated from the previous test. During the iterative testing process, the fuzzifier can autonomously accumulate experience and adjust strategies based on the feedback obtained from the fuzz testing process. In this embodiment, before the start of the Nth test, the fuzzifier has explored N-1 times.

[0133] like Figure 7As shown in the figure, before each round of attempts, the fuzzer will analyze based on the accumulated experience to determine which part of the message is more valuable to mutate, and give priority to the corresponding part for mutation. The fuzzer uses the accumulated experience to conduct tests more effectively, increase the proportion of valid test cases, reduce the generation of redundant test cases, and improve the ability to discover security vulnerabilities.

[0134] Based on segmented mutation, the Thompson sampling algorithm can calculate the sampling probability of each segment by the cumulative number of times the reward value is added with "1" and the reward value is added with "0" during the fuzzy test, using the β distribution probability density function, and select the best segment for mutation.

[0135] Each seed sequence will have a threshold for the number of test rounds. When the number of test rounds reaches the threshold, the next seed sequence will be selected for testing.

[0136] The flowchart of the fuzzy test based on the seed sequence provided by the embodiment of the present invention is as follows: Figure 5 As shown, the messages in the seed sequence are divided into three categories, namely, guide messages, target messages and remaining messages.

[0137] The seed sequence is expressed as formula (1):

[0138] (1);

[0139] In the formula, Seed represents the complete seed sequence, msg1 represents the guide message, msg2 represents the target message, and msg3 represents the remaining messages.

[0140] The seed sequence after mutation is expressed as formula (2):

[0141] (2);

[0142] In the formula, Seed mutated Represents the mutated seed sequence, and mutated (msg2) represents the mutated target message.

[0143] As shown in formula (1), msg1 is the guide message used to guide the seed sequence to the target state. As shown in formula (2), after the seed sequence is guided to the target state, the target message msg2 is mutated, and the mutated target message mutated (msg2) is sent to the target program. After the mutated message is sent, in order to make full use of the captured message, the remaining message msg3 is also sent to the target program.

[0144] For example, in the test of FTP server, the boot message msg1 is "USER test", the target message msg2 is "PASS testpass", and the remaining message msg3 includes "XMKD", "NLIST" and "QUIT". The boot message msg1 guides the service program to "331 Please specify the password", and the test program mutates the target message msg2 "PASS testpass". After sending the mutated message mutated (msg2), the server enters the "530 Please login with USERand PASS" state. Then send the remaining message msg3. Sending "XMKD" and "NLIST" does not cause the state to change. After sending "QUIT", the server responds with "221 Goodbye.". Finally, send a reset message to reset the service program for the next round of testing.

[0145] In summary, the method of the present invention adopts Thompson sampling as the benchmark strategy for segment mutation. Thompson sampling is based on the Bayesian principle and has strong real-time performance. The prior distribution is mainly obtained based on the experience accumulated in historical fuzzy testing, and the posterior distribution is updated based on the prior distribution information of each segment. Sampling is performed based on the posterior distribution to obtain a value that conforms to the distribution, that is, the sampling value, and the segment that needs to be mutated is selected based on the sampling value.

[0146] The above distribution-based selection method is highly adaptive. When selecting segments, it is possible to achieve a balance between segments that have already received higher rewards and segments that have not yet been selected. Intuitively, when the selection of certain segments receives higher rewards, the posterior distribution of these segments will change positively, and the probability of being selected will also increase.

[0147] In the above steps, when a new state of the test target is triggered, the new state of the test target and the triggering message are added to the state model, and the message sequence triggering the new state is added to the seed pool as a new seed sequence.

[0148] Step 3: Use the test cases generated by message mutation to test the test target.

[0149] Example 2

[0150] A computer-readable storage medium having computer instructions stored thereon, characterized in that the computer instructions implement the steps of the method in the above-mentioned embodiment 1 when executed by a processor.

[0151] Example 3

[0152] The present invention also provides a computer program product, including computer instructions, characterized in that when the computer instructions are executed by a processor, the steps of the method in the above-mentioned embodiment 1 are implemented.

[0153] In summary, the present implementation involves two core strategies: a sampling strategy and a reward updating strategy. The sampling strategy and the reward updating strategy are interconnected and work together.

[0154] The sampling strategy is mainly used to guide the selection of segments, that is, to select the segments that need to be mutated based on the current accumulated experience. Segment selection is implemented based on the β distribution probability density function, which characterizes the probability of a segment being selected by using the β distribution probability density function. The β distribution probability density function is suitable for situations where parameters are uncertain. This embodiment regards segment selection as a selection problem based on probability distribution. A segment corresponds to a parameter, which is sampled based on the β distribution and used as the basis for segment selection. The larger the parameter value, the higher the probability that the segment will be selected. As the fuzz test proceeds, test cases are constructed by mutating the selected segments. After being sent to the test target, the reward values ​​are collected and fed back to the fuzzifier. The fuzzifier will use the reward value as experience to synchronize the β distribution and continue to guide the selection of segments with the β distribution.

[0155] The core of the reward update strategy is to adjust the rewards of each segment according to the state change of the test target, and after selecting the segment that needs to be mutated, replace the data at the selected segment with the mutated data to build a new test case, send the new test case to the test target, and update the reward value of the segment that mutated in the current step according to the response of the test target. Based on the state coverage principle, if the test case triggers a new state, the reward value of the segment that mutated in the current step is added with "1", otherwise it is added with "0". It aims to explore new states by continuously mutating the segment data in the message, and dynamically adjust the reward value of each segment in the message according to whether the mutation of a segment can trigger the new state of the test target, thereby improving the test efficiency and test effect.

[0156] The present invention aims to optimize state selection and message mutation during fuzz testing through an innovative fuzz testing method guided by protocol state coverage, thereby improving the efficiency of protocol fuzz testing. The core of the protocol state coverage-oriented method is to intelligently enable fuzz testing to focus on protocol states and message segments that may reveal new states of the test target when performing state selection and message segment mutation. The present invention can not only effectively reduce the number of invalid tests and avoid waste of resources, but also significantly improve the overall efficiency of the test, ensuring that the test process is more focused on key protocol states and message segments.

[0157] The present invention proposes an innovative stateful protocol fuzz testing method based on reinforcement learning, which deeply integrates reinforcement learning theory and fuzz testing technology. By introducing the reinforcement learning mechanism, combining the state selection strategy of the UCB algorithm with the segmented mutation technology of the Thompson sampling algorithm, the intelligence and efficiency of the stateful protocol fuzz testing are significantly improved. The method provided by the present invention can not only guide the test to cover the new state of the test target, but also feedback the test effect through the reward mechanism, thereby realizing the precise mutation and optimization of the test case. The testing process of the present invention greatly reduces the generation of redundant test cases, making the fuzz testing process more efficient and targeted. At the same time, the dynamic update of the state model and the expansion of the seed pool ensure the comprehensive coverage of the test, effectively improve the depth and breadth of the fuzz test, and can provide strong technical support for vulnerability mining.

[0158] The present invention effectively solves the problem of blind mutation and low efficiency in traditional fuzz testing. By intelligently evaluating candidate states and implementing targeted mutation operations based on UCB values, the exploration capability of fuzz testing is significantly enhanced, and the effectiveness of testing is improved. At the same time, according to the state transition of the test target, the test strategy is dynamically adjusted to achieve continuous optimization of the test process, providing new ideas for the security testing of complex stateful protocols and laying a solid foundation for improving the overall security and reliability of software systems.

[0159] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A stateful protocol fuzz testing method based on reinforcement learning, characterized in that: include: Capture the network traffic generated by the test target during communication; Input the network traffic generated by the test target during the communication process into a state model containing message information, extract a seed sequence including the state information and message content, and output the seed sequence; Mutating the seed sequence to generate a test case for testing the test target; The method of mutating the seed sequence to generate a test case includes: Input the seed sequence into the fuzzifier, evaluate candidate states of the test target and select a target state for the current test from the candidate states; By sending the guide message in the seed sequence, the test target is guided to the target state; The message corresponding to the selected target state is divided into multiple segments according to special symbols; Based on the Thompson sampling algorithm, a segment is selected from the multiple segments to perform segment mutation, so as to obtain a mutated segment and an unmutated segment, and the mutated segment and the unmutated segment are spliced ​​to generate a test case; Send the test case to the test target for testing, and repeat the following steps until the preset test termination condition is reached: When the state of the test target changes, the reward mechanism is triggered and the reward value is accumulated; Use the fuzzifier to update the sampling probability of the segment that has mutated in the current step according to the accumulation of reward values; Select a segment with the highest sampling probability for mutation, and concatenate the mutated segment and the unmutated segment in the current step to construct a new test case; Send new test cases to the test target for testing and monitor the status changes of the test target; When the state of the triggering test target changes to a new state, the new state of the test target and the triggering message are added to the state model containing message information for updating, and the message sequence triggering the new state is added to the seed pool as a new seed sequence.

2. The stateful protocol fuzz testing method based on reinforcement learning according to claim 1 is characterized in that: The method for constructing the state model containing message information comprises: Capturing network traffic generated by the test target during each communication process, and extracting data packets from the network traffic; According to the source IP address and the destination IP address of the data packet, the data packet is divided into a server response message and a client service request message; Extracting keywords from the server response message to represent the state of the server and establishing an initial state model based on the keywords, wherein the keywords include a state code and text content; Extract the application layer payload of the client service request message, perform mutation processing to obtain the mutated payload, encapsulate the mutated payload back into the client service request message data packet, and send the encapsulated client service request message data packet to the server for testing; The client service request message and the initial state model are merged to obtain a state model containing message information.

3. The stateful protocol fuzz testing method based on reinforcement learning according to claim 2 is characterized in that: The initial state model and the state model containing message information are stored in a graph structure. Each node in the graph structure represents a state of the test target, and the edges between the nodes represent the messages that cause the test target state to change. When a new state appears in the test target, the new state is inserted into the graph structure as a new node.

4. The stateful protocol fuzz testing method based on reinforcement learning according to claim 1 is characterized in that: The method for selecting the target state includes: Evaluate the candidate states of the test target based on the UCB algorithm and calculate the UCB value of each candidate state; The candidate state with the highest UCB value is taken as the target state for the current test.

5. The stateful protocol fuzz testing method based on reinforcement learning according to claim 1 is characterized in that: The method of dividing the message corresponding to the selected target state into multiple segments according to special symbols includes: When there are special symbols in the message, the message is divided according to the special symbols in the message to obtain multiple segments; When there is no special symbol in the message, the message is divided into multiple segments by using a fixed-length division method.

6. The stateful protocol fuzz testing method based on reinforcement learning according to claim 1 is characterized in that: When the state of the test target changes, the reward mechanism is triggered, the reward value is accumulated, and the fuzzifier is used to update the sampling probability of the segment that has mutated in the current step according to the accumulation of the reward value; wherein the rule of the reward mechanism is: When the trigger test target changes to a new state, add "1" to the reward value of the segment where the current mutation occurs; When the trigger test target changes to a known state, add "0" to the reward value of the segment where the current mutation occurs.

7. The stateful protocol fuzz testing method based on reinforcement learning according to claim 6 is characterized in that: The rules of the reward mechanism also include: When the test target enters the exception handling state, add "0" to the reward value of the segment where the mutation occurs. When the protocol communication is based on the C / S architecture, the reward value of the segment that is currently mutated in the message is dynamically updated according to the real-time response of the server to the client request.

8. The stateful protocol fuzz testing method based on reinforcement learning according to claim 7 is characterized in that: The method for accumulating the reward value includes: Repeat the following steps until the number of test times for the message corresponding to the currently selected target state reaches the test termination condition: Record the cumulative number of times the reward value is increased by "1" and the reward value is increased by "0"; According to the cumulative number of times the reward value is added with "1" and the reward value is added with "0", the sampling probability of each segment is calculated using the beta distribution probability density function.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images