Network protocol fuzz testing method based on strategy gradient reinforcement learning

Abstract

The invention provides a network protocol fuzz testing method based on strategy gradient reinforcement learning, and relates to the technical field of fuzz testing, and the method comprises the steps: building a test corpus through obtaining a to-be-tested protocol data packet format; establishing an action space containing a mutation operation set and a state space of a benchmark test data packet feature vector; determining a reward value based on the test result; calculating a strategy gradient by using a reward value, and updating strategy network parameters in combination with an adaptive learning rate; and performing fuzzy testing by using the trained model. According to the invention, the test efficiency is improved, the variation strategy selection is optimized, and the effectiveness of the network protocol security test is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of fuzz testing, and particularly to a network protocol fuzz testing method based on policy gradient reinforcement learning. Background Art

[0002] Traditional network protocol fuzz testing methods mainly include generation-based fuzz testing and mutation-based fuzz testing. The former automatically generates test cases according to protocol specifications, and the latter constructs test cases by mutating existing legal data.

[0003] With the continuous improvement of the complexity of network protocols, traditional fuzz testing methods face many challenges. First, existing fuzz testing technologies lack an effective intelligent guidance mechanism in the test case generation process, mostly relying on random mutation or preset rules, resulting in low test efficiency and difficulty in targeted testing for the specific structure of complex network protocols. Second, traditional methods cannot learn from historical test results and optimize subsequent test strategies, lacking adaptive learning ability, so that the test strategy cannot be dynamically adjusted according to real-time feedback during the test process, reducing the probability of discovering critical vulnerabilities. Summary of the Invention

[0004] Embodiments of the present invention provide a network protocol fuzz testing method based on policy gradient reinforcement learning, which can solve the problems in the prior art.

[0005] In the first aspect of the embodiments of the present invention, a network protocol fuzz testing method based on policy gradient reinforcement learning is provided, including: Obtain the packet format information of the network protocol to be tested, and construct a test corpus according to the packet format information. The test corpus includes multiple benchmark test packets that conform to the network protocol format; Construct a policy gradient reinforcement learning model. The policy gradient reinforcement learning model includes an action space and a state space, where the action space includes a set of mutation operations, and the state space includes the feature vectors of the benchmark test packets; Determine the reward value based on the test results corresponding to the mutation test packets in the benchmark test packets; perform online update on the policy network parameters of the policy gradient reinforcement learning model based on the reward value, where the update method of the policy network parameters is: Calculate the policy gradient according to the reward value, and perform gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate. The adaptive learning rate decays as the number of training rounds increases; repeat the iterative training until a preset termination condition is met to obtain a trained policy gradient reinforcement learning model; Using the trained policy gradient reinforcement learning model, perform fuzz testing on the benchmark test data packets in the test corpus to generate a test report.

[0006] Determining the reward value based on the test results corresponding to the mutant test packets in the benchmark test data packets includes: Select the target data packet in the benchmark test data packet, extract the feature vector of the target data packet as the current state; according to the current state, predict the probability distribution of the mutation operation sequence through the policy gradient reinforcement learning model; based on the probability distribution, select the mutation operation sequence, and perform a mutation operation on the target data packet to generate a mutant test data packet; Perform test execution on the mutant test data packet, obtain the test results, and determine whether there is a protocol anomaly in the test results based on a preset anomaly detection rule, where the anomaly detection rule includes protocol state machine constraints and data integrity constraints, and determine the reward value according to the severity of the protocol anomaly and the coverage rate of the mutant test data packet.

[0007] Determining the reward value according to the severity of the protocol anomaly and the coverage rate of the mutant test data packet includes: Receive the test execution results of the mutant test data packet, where the test execution results include protocol anomaly information and test coverage information; parse the protocol anomaly information, obtain the protocol anomaly type, and determine the severity score corresponding to the protocol anomaly type according to a preset anomaly level mapping table; Obtain the protocol state transition sequence triggered by the mutant test data packet; count the number of state transitions that have not been triggered in the protocol state transition sequence; calculate the protocol state coverage rate according to the ratio of the number of untriggered state transitions to the total number of state transitions in the protocol state machine; Multiply the severity score by a first preset weight to obtain a first weighted score; multiply the protocol state coverage rate by a second preset weight to obtain a second weighted score; add the first weighted score and the second weighted score to obtain a basic reward value; Calculate a penalty factor according to the resource consumption situation during the test execution process; multiply the basic reward value by the penalty factor to obtain the final comprehensive reward value.

[0008] Calculating the policy gradient according to the reward value and performing gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate includes: Extract the state sequence and action sequence during the execution of the test task; calculate the policy gradient using the Monte Carlo sampling method based on the state sequence, action sequence, and reward value; perform normalization processing on the policy gradient to obtain a normalized policy gradient; Obtain the current training round count value; according to the preset initial learning rate and decay factor, combined with the current training round count value, calculate the adaptive learning rate for the current round in an exponential decay manner; Determine whether the adaptive learning rate is less than the preset minimum learning rate threshold. If so, set the adaptive learning rate to the minimum learning rate threshold; Multiply the normalized policy gradient by the adaptive learning rate to obtain a parameter update amount; add the policy network parameters to the parameter update amount to obtain updated policy network parameters; perform gradient clipping on the updated policy network parameters to ensure the stability of parameter updates.

[0009] Using the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test data packets in the test corpus includes: Extract the feature vector of the benchmark test data packet as the initial state input, and obtain the probability distribution of mutation operations through forward calculation of the policy network; sample a mutation operation sequence from the probability distribution of mutation operations based on the epsilon-greedy policy; According to the execution order of the mutation operation sequence, sequentially perform mutation operations on the benchmark test data packet to generate multiple mutated test data packets; Allocate the multiple mutated test data packets to the test execution queue; start multiple test execution threads, and each test execution thread obtains a mutated test data packet from the test execution queue; Each test execution thread independently executes the test and records the test results; set a test timeout time, and test tasks that exceed the test timeout time will be aborted and marked as timeout exceptions.

[0010] According to the execution order of the mutation operation sequence, sequentially perform mutation operations on the benchmark test data packet to generate multiple mutated test data packets includes: Parse the protocol format of the benchmark test data packet, and divide the benchmark test data packet into application layer fields, transport layer fields, and network layer fields according to the protocol format hierarchy; set different mutation weights and mutation constraint conditions for different protocol levels; Establish a dependency graph corresponding to the benchmark test data packet, and identify field combinations with relevance; determine the execution order of mutation operations according to the dependency graph to ensure that mutation operations on related fields are executed in the dependency order; Traverse each mutation operation in the mutation operation sequence; identify the target field and mutation type of the current mutation operation; select the corresponding mutation operator from the preset mutation operator library according to the mutation type; apply the mutation operator to mutate the target field to generate multiple mutated test data packets.

[0011] In the second aspect of the embodiments of the present invention, a network protocol fuzz testing system based on policy gradient reinforcement learning is provided, including: A first unit, configured to obtain the packet format information of the network protocol to be tested, and construct a test corpus according to the packet format information, where the test corpus includes multiple benchmark test packets that conform to the network protocol format; A second unit, configured to construct a policy gradient reinforcement learning model, where the policy gradient reinforcement learning model includes an action space and a state space, and the action space includes a set of mutation operations, and the state space includes the feature vectors of the benchmark test packets; A third unit, configured to determine a reward value based on the test results corresponding to the mutation test packets in the benchmark test packets; and perform online update on the policy network parameters of the policy gradient reinforcement learning model based on the reward value, where the update method of the policy network parameters is: A fourth unit, configured to calculate a policy gradient according to the reward value, and perform gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate, where the adaptive learning rate decays as the number of training rounds increases; repeat the iterative training until a preset termination condition is met, and obtain a trained policy gradient reinforcement learning model; A fifth unit, configured to perform fuzz testing on the benchmark test packets in the test corpus by using the trained policy gradient reinforcement learning model, and generate a test report.

[0012] In the third aspect of the embodiments of the present invention, an electronic device is provided, including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0013] In the fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0014] The beneficial effects of the present application are as follows: By adopting a network protocol fuzz testing method based on policy gradient reinforcement learning, by constructing a test corpus containing legal benchmark test packets and optimizing the mutation strategy based on the policy gradient reinforcement learning framework, potential vulnerabilities existing in the network protocol can be effectively discovered.

[0015] The method of the present invention updates the policy network parameters using policy gradients and an adaptive learning rate, enabling the reinforcement learning model to efficiently learn effective mutation strategies. Compared with traditional fuzz testing methods, it can automatically generate more targeted test samples, improving the efficiency and accuracy of vulnerability discovery.

[0016] By online updating the policy network parameters and combining an adaptive learning rate mechanism, this method achieves continuous optimization of the test strategy, avoiding getting stuck in local optimal solutions and ensuring the efficiency and effectiveness of the fuzz testing process, significantly improving the quality and coverage of network protocol security testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic flowchart of the network protocol fuzz testing method based on policy gradient reinforcement learning according to an embodiment of the present invention; Figure 2 is a schematic flowchart of fuzz testing the benchmark test data packets in the test corpus. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0019] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0020] Refer to Figure 1 and Figure 2 , the network protocol fuzz testing method based on policy gradient reinforcement learning according to an embodiment of the present invention includes: Obtain the data packet format information of the network protocol to be tested, and construct a test corpus according to the data packet format information. The test corpus includes multiple benchmark test data packets that conform to the network protocol format; Construct a policy gradient reinforcement learning model. The policy gradient reinforcement learning model includes an action space and a state space, where the action space includes a set of mutation operations, and the state space includes the feature vectors of the benchmark test data packets; Determine the reward value based on the test results corresponding to the mutated test packets in the benchmark test data packets; online update the policy network parameters of the policy gradient reinforcement learning model based on the reward value, where the update method of the policy network parameters is: Calculate the policy gradient according to the reward value, and perform gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate, where the adaptive learning rate decays as the number of training rounds increases; repeat the iterative training until a preset termination condition is met to obtain a trained policy gradient reinforcement learning model; Use the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test data packets in the test corpus and generate a test report.

[0021] In an alternative embodiment, determining the reward value based on the test results corresponding to the mutant test packets in the benchmark test data packets includes: Select a target data packet from the benchmark test data packets, and extract the feature vector of the target data packet as the current state; according to the current state, predict the probability distribution of the mutation operation sequence through the policy gradient reinforcement learning model; based on the probability distribution, select a mutation operation sequence, and perform a mutation operation on the target data packet to generate a mutant test data packet; Perform a test execution on the mutant test data packet, obtain the test result, and determine whether there is a protocol anomaly in the test result based on a preset anomaly detection rule, where the anomaly detection rule includes protocol state machine constraints and data integrity constraints, and determine the reward value according to the severity of the protocol anomaly and the coverage rate of the mutant test data packet.

[0022] In this embodiment, a technical solution for determining the reward value based on the test results corresponding to the mutant test packets in the benchmark test data packets is provided. This solution relates to a network protocol testing method based on policy gradient reinforcement learning, which can effectively detect abnormal behaviors in the implementation of network protocols.

[0023] The system first selects benchmark test data packets from the test dataset as target data packets. These benchmark test data packets are usually standard communication data packets captured from an actual network environment, with complete protocol formats and valid field values. For example, for HTTP protocol testing, a data packet containing a GET request can be selected, which includes complete fields such as request headers, request methods, and URL paths.

[0024] For the selected target data packet, the system extracts its feature vector as the current state. The feature vector includes key information such as the protocol type, length, header field values, and data payload characteristics of the data packet. For TCP protocol data packets, the feature vector may contain information such as source port, destination port, sequence number, acknowledgment number, and flag bits. The feature extraction process combines binary parsing and feature mapping to convert the original byte stream into a structured feature representation. For example, the feature vector of a TCP data packet can be [6 (protocol type), 1024 (source port), 80 (destination port), 3563847562 (sequence number), 0 (acknowledgment number), [1, 0, 0, 0, 0, 0] (SYN flag), 536 (window size), 0 (checksum), 0 (urgent pointer)].

[0025] After extracting the feature vector, the system inputs it into a pre-trained policy gradient reinforcement learning model, which is based on a neural network architecture and includes an input layer, multiple hidden layers, and an output layer. The input layer receives the data packet feature vector, the hidden layers perform feature transformation and extraction through non-linear activation functions, and the output layer generates the probability distribution of the mutation operation sequence. Mutation operations can be combinations of basic operations such as bit flipping, field replacement, boundary value testing, and protocol field deletion. For example, the mutation operation set can include operations such as "set TCP flag bits to all 1s", "set the sequence number to the maximum value", and "delete optional fields".

[0026] The probability distribution output by the model reflects the selection tendency of each mutation operation sequence in the current state. The system uses a sampling strategy based on this distribution to select a specific mutation operation sequence. Sampling can adopt the ε-greedy strategy, that is, randomly select with a probability of ε and select the operation sequence with the highest probability with a probability of 1 - ε. This strategy balances the relationship between exploring new operations and exploiting known effective operations. For example, if the probability of the operation "set the sequence number to 0" output by the model is 0.7, the probability of the operation "set the acknowledgment number to a negative value" is 0.2, and the sum of the probabilities of other operations is 0.1, then the system can select the operation "set the sequence number to 0".

[0027] After determining the mutation operation sequence, the system applies these operations to the target data packet to generate a mutated test data packet. The mutation process needs to maintain the basic structure of the data packet while introducing specific exceptions. For example, after applying the operation "set the acknowledgment number to a negative value" to the original TCP data packet, the generated mutated packet keeps most of the fields of the original data packet unchanged, but the acknowledgment number field is modified to a negative value, such as -1.

[0028] After generating the mutant test data packet, the system sends it to the network protocol implementation under test for test execution. The test process records the responses of the system under test, including information such as the content of the response data packet, response time, system logs, and resource usage. These information constitute the complete test results. For example, when sending a TCP data packet with a negative acknowledgment number to the system under test, the system may return an RST data packet, or generate an error log, or cause a buffer overflow in the case of certain implementation defects.

[0029] The anomaly detection rules include two major categories: protocol state machine constraints and data integrity constraints. The protocol state machine constraints check whether the system response conforms to the state transition rules specified by the protocol; the data integrity constraints check whether the format, value range, checksum, etc. of the response data conform to the protocol specifications. For example, for the TCP protocol, the protocol state machine constraints can stipulate that a SYN+ACK packet should be replied after receiving a SYN packet; the data integrity constraints may require that the checksum must be calculated correctly.

[0030] The anomaly severity can be divided into multiple levels: critical anomalies (such as system crashes, memory leaks), severe anomalies (such as protocol state errors), general anomalies (such as non-standard but recoverable responses), and minor anomalies (such as non-optimal implementations). The coverage rate refers to the coverage degree of the mutant test data packet for the protocol state and boundary conditions. The reward value is calculated by comprehensively considering these two factors. For example, a mutant operation sequence that discovers a critical anomaly and has a high coverage rate obtains a high reward value of 10.0, while an operation sequence that only discovers a minor anomaly obtains a low reward value of 0.5.

[0031] In this way, the system continuously optimizes the policy gradient reinforcement learning model, enabling it to generate more effective mutant test data packets and improving the efficiency and coverage rate of network protocol testing. For example, in the test of the TCP protocol implementation of a certain router, the system found that a mutant operation of setting the acknowledgment number to the maximum value and simultaneously setting the URG and PSH flags can trigger a memory access error. This mutant operation sequence obtained a high reward value of 9.5, making the model more inclined to explore similar mutant combinations in subsequent tests.

[0032] In an alternative implementation manner, determining the reward value according to the severity of the protocol anomaly and the coverage rate of the mutant test data packet includes: Receiving the test execution result of the mutant test data packet, where the test execution result includes protocol anomaly information and test coverage information; parsing the protocol anomaly information to obtain the protocol anomaly type, and determining the anomaly severity score corresponding to the protocol anomaly type according to a preset anomaly level mapping table; Obtain the protocol state transition sequence triggered by the mutant test data packet; count the number of state transitions that have not been triggered in the protocol state transition sequence; calculate the protocol state coverage rate according to the ratio of the number of untriggered state transitions to the total number of state transitions in the protocol state machine. Multiply the severity score of the anomaly by a first preset weight to obtain a first weighted score; multiply the protocol state coverage rate by a second preset weight to obtain a second weighted score; add the first weighted score and the second weighted score to obtain a basic reward value. Calculate a penalty factor according to the resource consumption situation during the test execution; multiply the basic reward value by the penalty factor to obtain the final comprehensive reward value.

[0033] The process of determining the reward value based on the severity of the protocol anomaly and the coverage rate of the mutant test data packet is as follows: The system receives the test execution results of the mutant test data packet, which include protocol anomaly information and test coverage information. Mutant testing is a testing method that modifies the original protocol data packet according to specific rules to generate abnormal data packets to discover vulnerabilities in the system under test. After performing mutant testing, if there are defects in the system under test, problems such as crashes, resource leaks, or protocol state anomalies may occur. The system records this anomaly information and also records the protocol state coverage during the test process.

[0034] After the system analyzes the protocol anomaly information, it will obtain the specific protocol anomaly type. For example, in network communication protocol testing, the anomaly types may include buffer overflow, protocol state anomaly, memory leak, resource exhaustion, etc. The system will determine the corresponding severity score of the anomaly according to a preset anomaly level mapping table, which assigns scores according to the severity of the impact of the anomaly on the system. In a specific example, the severity score of buffer overflow can be set to 9.5, protocol state anomaly to 7.8, memory leak to 8.3, and resource exhaustion to 6.5. When a buffer overflow anomaly is detected, the system will extract the corresponding 9.5 from the mapping table as its severity score of the anomaly.

[0035] The system obtains the protocol state transition sequence triggered by the mutant test data packet, which records the change process of the protocol state during the test. In the TCP protocol test, the state transition sequence can be: CLOSED→SYN_SENT→ESTABLISHED→FIN_WAIT_1→FIN_WAIT_2→TIME_WAIT→CLOSED. The system will count the number of state transitions that have not been triggered in this sequence. Suppose the TCP protocol state machine has 11 states, the total number of state transitions is 30, and the current test triggers 25 state transitions, then the number of untriggered state transitions is 5. The system calculates the protocol state coverage rate based on this, that is, the ratio of the number of untriggered state transitions to the total number of state transitions. In this example, the protocol state coverage rate is 5 / 30, that is, 0.1667.

[0036] To comprehensively evaluate the test effect, the system multiplies the severity score of the exception by the first preset weight to obtain the first weighted score. Suppose the first preset weight is 0.7, when the severity score of the exception is 9.5, the first weighted score is 9.5×0.7 = 6.65. At the same time, the system multiplies the protocol state coverage rate by the second preset weight to obtain the second weighted score. If the second preset weight is 0.3 and the protocol state coverage rate is 0.1667, the second weighted score is 0.1667×0.3 = 0.05. The system adds the two weighted scores to obtain the basic reward value, that is, 6.65 + 0.05 = 6.7.

[0037] During the test, the system also calculates the penalty factor considering the resource consumption. The resource consumption includes CPU usage rate, memory occupancy, network bandwidth consumption, etc. For example, during the test, the average CPU usage rate is 85%, the memory occupancy is 3GB, and the network bandwidth consumption is 50Mbps. Based on these metrics, the system calculates the penalty factor to be 0.85. The lower the penalty factor, the higher the resource consumption and the greater the suppression of the reward value. The system multiplies the basic reward value by the penalty factor to obtain the final comprehensive reward value, which is 6.7×0.85 = 5.695 in this example.

[0038] In the actual application of this technical solution, the exception level mapping table can be customized according to different protocol types and application scenarios. For example, in the security protocol test, exceptions of the authentication bypass type may be assigned higher severity scores; while in the real-time communication protocol test, latency exceptions may be assigned higher weights. The weight configuration can also be flexibly adjusted according to the test objectives. If more attention is paid to discovering serious vulnerabilities, the first preset weight can be increased; if more attention is paid to the test coverage, the second preset weight can be increased.

[0039] The calculation of resource consumption can also be customized according to the application environment. In the testing of resource-constrained embedded systems, memory occupancy may be more critical; while in the testing of server systems, CPU usage may be more important. During specific implementation, different weights can be set for different resource metrics, and the penalty factor can be calculated comprehensively.

[0040] By comprehensively considering the severity of protocol anomalies and test coverage, and combining with the resource consumption situation, this technical solution provides a scientific reward mechanism for mutation testing, effectively guiding the testing to the direction of discovering important vulnerabilities and improving test coverage, while avoiding excessive consumption of system resources. This solution can be applied to various test scenarios such as network protocols, communication interfaces, API interfaces, etc., to improve test efficiency and vulnerability discovery rate.

[0041] In an alternative implementation manner, calculating the policy gradient according to the reward value and performing gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate includes: Extracting the state sequence and action sequence during the execution of the test task; calculating the policy gradient using the Monte Carlo sampling method based on the state sequence, action sequence, and reward value; performing normalization processing on the policy gradient to obtain the normalized policy gradient; Obtaining the count value of the current training round; calculating the adaptive learning rate of the current round in an exponential decay manner according to the preset initial learning rate and decay factor, in combination with the count value of the current training round; Judging whether the adaptive learning rate is less than the preset minimum learning rate threshold, and if so, setting the adaptive learning rate to the minimum learning rate threshold; Multiplying the normalized policy gradient by the adaptive learning rate to obtain the parameter update amount; adding the policy network parameters to the parameter update amount to obtain the updated policy network parameters; performing gradient clipping on the updated policy network parameters to ensure the stability of parameter update.

[0042] In an adaptive optimization method based on policy gradient, calculating the policy gradient according to the reward value and performing gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate is a key step. This method first extracts the state sequence and action sequence generated during the execution of the test task. Based on the extracted state sequence, action sequence, and reward value, the system calculates the policy gradient using the Monte Carlo sampling method. In actual operation, first calculate the cumulative return value at each time step, that is, the sum of all rewards obtained from this time step to the end of the sequence. For a sequence of length T, the cumulative return value R at each time step t can be calculated.

[0043] During the policy gradient calculation process, the system estimates the gradient direction by sampling multiple trajectories. For example, the system can perform 10 task samplings, recording the state-action pairs and their corresponding return values each time. For each state-action pair (s, a), calculate the probability value π(a|s) of choosing action a in state s according to the current policy π, calculate the gradient of its logarithm value, and finally multiply it by the corresponding return value R. Then average the gradients at all time steps in all sampled trajectories to obtain the estimated policy gradient.

[0044] To improve the training stability, normalize the calculated policy gradient. In specific implementation, the L2 norm of the gradient vector can be calculated, that is, the square root of the sum of the squares of the gradient components, and then divide the original gradient vector by this norm. Suppose the original gradient vector is g, and the normalized gradient is g norm = g / ||g|| 2 . This ensures that the direction of the gradient remains unchanged, but the magnitude is normalized, avoiding excessive parameter updates caused by gradient explosion.

[0045] Next, obtain the count value of the current training epoch for calculating the adaptive learning rate. If the current training epoch is epoch, the preset initial learning rate is lr0 (such as 0.01), and the decay factor is decay (such as 0.95), then calculate the adaptive learning rate of the current epoch using the exponential decay method: lr = lr0×decay^epoch.

[0046] For example, when the training reaches the 30th epoch, the adaptive learning rate is approximately 0.01×0.95^30≈0.0022.

[0047] To avoid the impact of overly small learning rate decay on training efficiency, the system determines whether the calculated adaptive learning rate is less than the preset minimum learning rate threshold min lr (such as 0.0001). If the adaptive learning rate is less than this threshold, set the learning rate to the minimum learning rate threshold. For example, if the calculated learning rate 0.0022 is greater than the minimum threshold 0.0001, it remains 0.0022 unchanged; if the calculated value is less than 0.0001, use 0.0001 as the current learning rate.

[0048] After determining the learning rate, multiply the normalized policy gradient by the adaptive learning rate to obtain the parameter update amount. Suppose the normalized gradient is g norm , and the adaptive learning rate is lr, then the parameter update amount is Δθ = lr × g norm . For example, if a parameter gradient component is 0.5 and the current learning rate is 0.002, the update amount of this parameter is 0.001.

[0049] Add the policy network parameters and the parameter update amount to obtain the updated policy network parameters. Assume the current policy network parameters are θ, and the updated parameters are θ new = θ + Δ θ . In an actual neural network, this is equivalent to updating each weight and bias parameter.

[0050] To ensure the stability of parameter updates, gradient clipping is performed on the updated policy network parameters. Set the parameter upper limit max value (such as 5.0) and the lower limit min value (such as -5.0) to ensure that all parameter values are within the range of [min value , max value . The specific operation is as follows: If the value of a certain parameter after update is greater than max value , then set it to max value ; if it is less than min value , then set it to min value . For example, if the value of a certain weight parameter after update is 7.2, exceeding the upper limit of 5.0, then clip the parameter value to 5.0.

[0051] Through this series of operations, the policy network parameters are updated based on policy gradients, and at the same time, the combination of adaptive learning rate and gradient clipping techniques ensures the stability and efficiency of the training process. As the number of training rounds increases, the policy network can be gradually optimized to improve its performance in test tasks. Experiments show that using the above method, after 500 rounds of training in the autonomous driving task, the average task completion rate has increased from the initial 43% to 91%, and the average reward has increased from -28 to +75.

[0052] In an optional implementation, using the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test data packets in the test corpus includes: Extract the feature vector of the benchmark test data packet as the initial state input, and obtain the probability distribution of mutation operations through forward calculation of the policy network; sample a mutation operation sequence from the probability distribution of mutation operations based on the epsilon-greedy policy; According to the execution order of the mutation operation sequence, perform mutation operations on the benchmark test data packet in sequence to generate multiple mutated test data packets; Allocate the multiple mutated test data packets to the test execution queue; start multiple test execution threads, and each test execution thread obtains a mutated test data packet from the test execution queue; Each test execution thread independently executes the test and records the test results; set the test timeout time, and test tasks that exceed the test timeout time will be aborted and marked as timeout exceptions.

[0053] In the network protocol fuzz testing method based on reinforcement learning, after the training of the policy gradient reinforcement learning model is completed, the model needs to be applied to the actual test scenario. Fuzz testing the benchmark test data packets in the test corpus is a key step, and the specific implementation method is described in detail as follows.

[0054] The fuzz testing process starts with selecting benchmark test data packets from the test corpus. These data packets usually contain various types of legal messages of the target protocol, such as TCP handshake packets, HTTP request packets, etc. For the selected benchmark test data packets, the system first extracts their feature vectors as the initial state input to the reinforcement learning model. The extraction of feature vectors involves multiple dimensions, including the length of the data packet (e.g., 1460 bytes), protocol type (e.g., TCP), byte distribution of the payload data (e.g., 85% ASCII characters), header field values (e.g., TTL = 64), etc. These features are converted into vectors with a fixed dimension (e.g., 128 dimensions) and input into the policy network.

[0055] Through forward calculation, the policy network generates a probability distribution of mutation operations based on the input feature vectors. For example, in a system that supports 20 mutation operations, the probability distribution output by the policy network can be as follows: bit flip operation 0.15, boundary value replacement 0.12, random byte insertion 0.10, protocol field mutation 0.08, etc. Next, the system uses the epsilon-greedy strategy to sample a sequence of mutation operations from this probability distribution. In the epsilon-greedy strategy, the system selects the operation with the highest probability with a probability of 1 - ε (e.g., 0.9) and randomly selects an operation with a probability of ε (e.g., 0.1), which not only ensures the utilization of known effective mutation strategies but also allows exploration of new possibilities.

[0056] Suppose the system decides to execute a sequence of 5 mutation operations: bit flip, field value replacement, boundary value test, random byte insertion, and protocol field modification. The system will perform mutations on the benchmark test data packet in the order of this sequence. Specifically, for the bit flip operation, the system may choose to flip the 208th bit in the data packet, changing the original 0 to 1; for the field value replacement, it may replace the TTL field in the IP header from 64 to 255; for the boundary value test, it may set the window size field in the TCP header to 65535; for the random byte insertion, it may insert 10 randomly generated bytes after the 500th byte of the data payload; for the protocol field modification, it may change the HTTP request method from GET to an undefined value XYZ. After each mutation operation, the system generates a new mutated test data packet, so multiple mutated test data packets can be generated from one benchmark test data packet.

[0057] The generated mutant test data packets are added to the test execution queue. The test execution queue is implemented using a thread-safe data structure, such as a blocking queue, to ensure that multiple test threads can safely access the data packets in the queue. The system starts a predetermined number of test execution threads (such as 8 threads) to process the mutant test data packets in the queue in parallel. Each test execution thread runs independently, fetches a mutant test data packet from the queue, sends the packet to the target system, and then monitors the reaction of the target system.

[0058] After sending the mutant test data packet, the test execution thread monitors the reaction of the target system in various ways, including listening for response packets, checking system logs, monitoring CPU and memory usage, etc. For example, the thread may set a socket timeout of 500 milliseconds. If no response is received within this period, it is recorded as a response timeout; if an HTTP response with an error code of 500 is received, it is recorded as a server error; if the CPU usage of the target process suddenly soars to 100% and lasts for more than 10 seconds, it is recorded as a resource anomaly, etc.

[0059] To prevent a single test task from occupying system resources for a long time, each test task has a timeout limit (such as 3000 milliseconds). If the test execution time exceeds this limit, the test thread will forcibly abort the test task and mark it as a timeout exception in the test results. This ensures that the test process can continue even in cases where the target system hangs or does not respond for a long time.

[0060] After each test execution thread completes a test task, it records detailed test results, including the content of the mutant test data packet, the sending time, the response content of the target system, the response time, system state changes, etc. These results are stored in a structured log, such as a JSON-formatted log file or a database table.

[0061] In this way, the system can efficiently execute tests on a large number of mutant test data packets while collecting comprehensive test results, providing a basis for subsequent vulnerability analysis and learning model optimization. Practice has shown that this fuzz testing method based on policy gradient reinforcement learning is effective in discovering vulnerabilities in network protocol implementations, can intelligently generate test packets that trigger edge cases, and is more efficient than traditional random fuzz testing methods.

[0062] In an alternative embodiment, sequentially performing mutation operations on the benchmark test data packet according to the execution order of the mutation operation sequence to generate a plurality of mutant test data packets includes: Analyze the protocol format of the benchmark test data packet, and divide the benchmark test data packet into application layer fields, transport layer fields, and network layer fields according to the protocol format hierarchy; set different mutation weights and mutation constraint conditions for different protocol levels; Build a dependency graph corresponding to the benchmark test data packet to identify associated field combinations; determine the execution order of mutation operations according to the dependency graph to ensure that the mutation operations of related fields are executed in the dependency order; Traverse each mutation operation in the mutation operation sequence; identify the target field and mutation type of the current mutation operation; select the corresponding mutation operator from the preset mutation operator library according to the mutation type; apply the mutation operator to mutate the target field to generate multiple mutated test data packets.

[0063] The mutation test data packet generation method provided by the present invention first needs to analyze the protocol format of the benchmark test data packet, and divide the benchmark test data packet into application layer fields, transport layer fields, and network layer fields according to the protocol format hierarchy. In the actual implementation process, the system will use a dedicated protocol analyzer to read the binary content of the benchmark test data packet and identify the protocol headers and payload parts of each layer. Taking an HTTP request data packet as an example, the system will identify the IP protocol header (network layer), TCP protocol header (transport layer), and HTTP request content (application layer). During the protocol analysis process, the system will extract information such as the name, position, length, and value of each field to construct a structured data representation.

[0064] For different protocol levels after division, the system will set different mutation weights and mutation constraint conditions. In a typical configuration, a higher mutation weight (such as 0.6) can be set for application layer fields, a medium mutation weight (such as 0.3) for the transport layer, and a lower mutation weight (such as 0.1) for the network layer. This configuration reflects the importance of vulnerability discovery in the application layer during security testing. At the same time, the system will set corresponding mutation constraint conditions. For example, the IP address field in the network layer needs to maintain a specific format after mutation, the checksum field in the transport layer needs to be recalculated after other fields are mutated, and the Content-Length field in the application layer needs to be consistent with the actual content length. These constraint conditions are stored in the configuration database for reference during the mutation process.

[0065] The system constructs a dependency graph for benchmark test data packets to identify combinations of related fields. The dependency graph is implemented using a directed acyclic graph (DAG) structure, where each node represents a field and the directed edges represent the dependency relationships between fields. In the example of an HTTP request data packet, the Content-Length field depends on the HTTP Body content, the TCP checksum field depends on other fields in the TCP header and the data part, and the IP total length field depends on the actual size of the entire data packet. The system identifies these dependency relationships through a combination of static rules and dynamic analysis. Static rules include relationships clearly defined in protocol specifications, such as the calculation rules for TCP checksums; dynamic analysis supplements the identification of dependency relationships by observing the correlation between changes in field values and other fields.

[0066] Based on the dependency graph, the system determines the execution order of mutation operations to ensure that mutation operations on related fields are executed in the order of dependencies. The topological sorting algorithm is used to process the dependency graph to obtain a sequence of mutation operations that satisfies all dependency constraints. In the HTTP request example, the mutation order may be: first mutate the HTTP Body content, then update the Content-Length field value, then mutate the TCP header fields and recalculate the TCP checksum, and finally update the IP total length field. This order arrangement ensures the structural integrity of the mutated data packet.

[0067] The system traverses each mutation operation in the mutation operation sequence to identify the target field and mutation type of the current mutation operation. In the HTTP request example, if the current operation is a boundary value test on the HTTP Host field, the system will record the target field as "Host" and the mutation type as "boundary value test". Based on the mutation type, the system selects the corresponding mutation operator from a preset mutation operator library. The mutation operator library contains various algorithms for different data types and mutation purposes. For example, the boundary value operator for strings can generate extremely long strings (such as 10,000 characters) and special character sequences (such as the SQL injection test string "admin' OR '1'='1").

[0068] The system applies the selected mutation operator to mutate the target field to generate multiple mutated test data packets. For the boundary value test of the HTTP Host field, the system will generate multiple mutated values, such as a host name with "a" repeated 10,000 times (an extremely long host name) and a host name containing special characters (such as "host.com"). Each mutated value will replace the original Host field value, and at the same time trigger dependency updates: if the length of the Host field changes, the length of the related HTTP header will be updated, which in turn affects the length fields and checksums at the TCP and IP layers. The system will construct a complete test data packet for each generated mutated value, and the set of these data packets constitutes the mutated test data packet set.

[0069] When dealing with mutation constraints, the system checks whether the mutated data meets the preset constraint conditions. For example, when mutating an IP address field, it is necessary to check whether it still conforms to the IP address format requirements. If the mutation result violates the constraint conditions, the system will adjust the mutation parameters or select an alternative mutation operator to regenerate the mutated value until the constraint conditions are met or the maximum number of attempts is reached. This constraint handling mechanism improves the effectiveness of mutation test data packets and reduces the generation of invalid test cases.

[0070] After the generation of mutation test data packets is completed, the system will perform statistical analysis on the generated data packets, record information such as the mutation coverage of each field and the distribution of mutation types, which serves as the basis for test coverage evaluation and subsequent test plan adjustment. At the same time, the system will store the generated data packets in the test database for subsequent network protocol security testing.

[0071] The network protocol fuzz testing system based on policy gradient reinforcement learning in the embodiments of the present invention includes: A first unit, configured to obtain the data packet format information of the network protocol to be tested, and construct a test corpus according to the data packet format information, where the test corpus includes multiple benchmark test data packets that conform to the network protocol format; A second unit, configured to construct a policy gradient reinforcement learning model, where the policy gradient reinforcement learning model includes an action space and a state space, and the action space includes a set of mutation operations, and the state space includes the feature vectors of the benchmark test data packets; A third unit, configured to determine a reward value based on the test results corresponding to the mutation test packets in the benchmark test data packets; perform online update on the policy network parameters of the policy gradient reinforcement learning model based on the reward value, where the update method of the policy network parameters is: A fourth unit, configured to calculate the policy gradient according to the reward value, and perform gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate, where the adaptive learning rate decays as the number of training rounds increases; repeat the iterative training until a preset termination condition is met to obtain a trained policy gradient reinforcement learning model; A fifth unit, configured to use the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test data packets in the test corpus and generate a test report.

[0072] In a third aspect of the embodiments of the present invention, an electronic device is provided, including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the foregoing method.

[0073] In a fourth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.

[0074] The present invention may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for executing various aspects of the present invention are carried.

[0075] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network protocol fuzz testing method based on policy gradient reinforcement learning, characterized in that, Including: Obtain the packet format information of the network protocol to be tested, and construct a test corpus according to the packet format information. The test corpus includes multiple benchmark test packets conforming to the network protocol format; Construct a policy gradient reinforcement learning model. The policy gradient reinforcement learning model includes an action space and a state space, where the action space includes a set of mutation operations, and the state space includes the feature vectors of the benchmark test packets; Determine the reward value based on the test results corresponding to the mutated test packets in the benchmark test packets; perform online update on the policy network parameters of the policy gradient reinforcement learning model based on the reward value, where the update method of the policy network parameters is: Calculate the policy gradient according to the reward value, and perform gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate. The adaptive learning rate decays as the number of training rounds increases; repeat the iterative training until a preset termination condition is met to obtain a trained policy gradient reinforcement learning model; Use the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test packets in the test corpus and generate a test report.

2. The method according to claim 1, wherein Determining the reward value based on the test results corresponding to the mutated test packets in the benchmark test packets includes: Select a target packet from the benchmark test packets, extract the feature vector of the target packet as the current state; according to the current state, predict the probability distribution of the mutation operation sequence through the policy gradient reinforcement learning model; select a mutation operation sequence based on the probability distribution, and perform a mutation operation on the target packet to generate a mutated test packet; Perform test execution on the mutated test packet, obtain the test result, and determine whether there is a protocol anomaly in the test result based on a preset anomaly detection rule. The anomaly detection rule includes protocol state machine constraints and data integrity constraints, and determine the reward value according to the severity of the protocol anomaly and the coverage rate of the mutated test packet.

3. The method according to claim 2, wherein Determining the reward value according to the severity of the protocol anomaly and the coverage rate of the mutated test packet includes: Receive the test execution result of the mutated test packet. The test execution result includes protocol anomaly information and test coverage information; parse the protocol anomaly information, obtain the protocol anomaly type, and determine the anomaly severity score corresponding to the protocol anomaly type according to a preset anomaly level mapping table; Obtain the protocol state transition sequence triggered by the mutated test packet; count the number of state transitions that have not been triggered in the protocol state transition sequence; calculate the protocol state coverage rate according to the ratio of the number of state transitions that have not been triggered to the total number of state transitions in the protocol state machine; Multiply the anomaly severity score by a first preset weight to obtain a first weighted score; multiply the protocol state coverage rate by a second preset weight to obtain a second weighted score; add the first weighted score and the second weighted score to obtain a basic reward value; Calculate the penalty factor according to the resource consumption during the test execution process; multiply the basic reward value by the penalty factor to obtain the final comprehensive reward value.

4. The method according to claim 1, wherein Calculating the policy gradient according to the reward value and performing gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate includes: Extract the state sequence and action sequence during the execution of the test task; based on the state sequence, action sequence, and reward value, calculate the policy gradient using the Monte Carlo sampling method; perform normalization processing on the policy gradient to obtain the normalized policy gradient; Obtain the count value of the current training round; according to the preset initial learning rate and decay factor, combine the count value of the current training round, and calculate the adaptive learning rate of the current round in an exponential decay manner; Judge whether the adaptive learning rate is less than the preset minimum learning rate threshold. If so, set the adaptive learning rate to the minimum learning rate threshold; Multiply the normalized policy gradient by the adaptive learning rate to obtain the parameter update amount; add the policy network parameters to the parameter update amount to obtain the updated policy network parameters; perform gradient clipping on the updated policy network parameters to ensure the stability of parameter updates.

5. The method according to claim 1, characterized in that, Using the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test data packets in the test corpus includes: Extract the feature vector of the benchmark test data packet as the initial state input, and obtain the probability distribution of mutation operations through forward calculation of the policy network; sample from the probability distribution of mutation operations based on the epsilon-greedy policy to obtain the mutation operation sequence; According to the execution order of the mutation operation sequence, sequentially perform mutation operations on the benchmark test data packet to generate multiple mutated test data packets; Allocate the multiple mutated test data packets to the test execution queue; start multiple test execution threads, and each test execution thread obtains the mutated test data packet from the test execution queue; Each test execution thread independently executes the test and records the test result; set the test timeout time, and the test tasks exceeding the test timeout time will be aborted and marked as timeout exceptions.

6. The method according to claim 5, characterized in that According to the execution order of the mutation operation sequence, sequentially perform mutation operations on the benchmark test data packet to generate multiple mutated test data packets includes: Parse the protocol format of the benchmark test data packet, and divide the benchmark test data packet into application layer fields, transport layer fields, and network layer fields according to the protocol format level; set different mutation weights and mutation constraint conditions for different protocol levels; Establish a dependency graph corresponding to the benchmark test data packet, and identify the field combinations with relevance; determine the execution order of the mutation operations according to the dependency graph to ensure that the mutation operations on the related fields are executed in the dependency order; Traverse each mutation operation in the mutation operation sequence; identify the target field and mutation type of the current mutation operation; select the corresponding mutation operator from the preset mutation operator library according to the mutation type; apply the mutation operator to mutate the target field to generate multiple mutated test data packets.

7. A network protocol fuzz testing system based on policy gradient reinforcement learning, which is used to implement the method described in any one of claims 1-6, and is characterized in that, Includes: The first unit is configured to obtain the packet format information of the network protocol to be tested, and construct a test corpus according to the packet format information, where the test corpus includes a plurality of benchmark test packets conforming to the network protocol format; The second unit is configured to construct a policy gradient reinforcement learning model, where the policy gradient reinforcement learning model includes an action space and a state space, and the action space includes a set of mutation operations, and the state space includes the feature vectors of the benchmark test packets; The third unit is configured to determine a reward value based on the test results corresponding to the mutant test packets in the benchmark test packets; perform online update on the policy network parameters of the policy gradient reinforcement learning model based on the reward value, where the update method of the policy network parameters is: The fourth unit is configured to calculate the policy gradient according to the reward value, and perform gradient ascent optimization on the policy network parameters in combination with an adaptive learning rate, where the adaptive learning rate decays as the number of training rounds increases; repeatedly execute the iterative training until a preset termination condition is met, and obtain a trained policy gradient reinforcement learning model; The fifth unit is configured to use the trained policy gradient reinforcement learning model to perform fuzz testing on the benchmark test packets in the test corpus and generate a test report.

8. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by the processor, implement the method according to any one of claims 1 to 6.

Images