Fuzzy test optimization model training method and device, fuzzy test optimization method and device, equipment and medium

Through the fuzz testing optimization model of reinforcement learning structure, the test case generation and strategy are optimized, and the problems of cumbersome and security risks of traditional fuzz testing are solved, and the efficiency and accuracy of vulnerability discovery of power systems are improved.

CN119065976APending Publication Date: 2024-12-03GUANGDONG POWER GRID CO LTD +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411157089.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Traditional fuzz testing methods rely on random or semi-random input generation methods, requiring artificial intervention to adjust strategies, the testing process is cumbersome, and there are loopholes in the power protocol in the power system, resulting in security risks.

Method used

A fuzzy testing optimization model with reinforcement learning structure is adopted to optimize the testing strategy through the policy network and the value network, test cases are generated, and tested in simulation and actual environments, and model parameters are adjusted based on feedback data to optimize the generation of test cases.

Benefits of technology

It improves the efficiency and accuracy of fuzz testing, reduces artificial intervention, enhances the ability to discover vulnerabilities in the power system, and reduces safety risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065976A_ABST
    Figure CN119065976A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a fuzzy test optimization model training and optimizing method and device, equipment and a medium. The fuzzy test optimization model training and optimizing method comprises the steps that a fuzzy test optimization model is established according to preset information; the preset information comprises a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model comprises a strategy network and a value network; training the fuzzy test optimization model by using a preset algorithm, and determining an optimization strategy; generating a test case according to the optimization strategy; and testing the test case in the simulation environment and the actual environment to obtain test feedback data, and adjusting parameters of the fuzzy test optimization model according to the test feedback data. According to the technical scheme, the fuzzy test optimization model is trained by using the preset algorithm, the optimization strategy is determined, the test case is generated according to the optimization strategy, and the parameters of the fuzzy test optimization model are adjusted according to the test case, so that the generation of the test case is optimized, and the fuzzy test efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of fuzz testing technology, and in particular to a training and optimization method, device, equipment and medium for a fuzz testing optimization model. Background Art

[0002] The power protocol in industrial control systems refers to the communication protocol used to monitor, manage and control the power system. The power system is a large and complex system, including power production, transmission and distribution. In order to monitor and control the various components of the power system, specific communication protocols are required to ensure that information can be effectively exchanged between devices. However, with the continuous development of network technology, the power system is also facing new security challenges, such as cyber attacks, malware, etc.

[0003] Commonly used power protocols in power systems include IEC61850, Modbus, DNP3, etc. These protocols have some vulnerabilities, such as buffer overflow, unauthenticated access, plain text communication, protocol parsing problems, etc. These vulnerabilities may be exploited by hackers or attackers, resulting in security risks in the power system. Therefore, it is crucial to perform fuzz testing on the power system and monitor the power to ensure the robustness of the power system and prevent potential threats.

[0004] At present, traditional fuzz testing usually relies on random or semi-random input generation methods and requires human intervention to adjust strategies and parameters, making the testing process rather cumbersome. Summary of the invention

[0005] The embodiments of the present disclosure provide a training and optimization method, device, equipment and medium for a fuzz test optimization model, which optimizes the generation of test cases and improves the efficiency and accuracy of fuzz testing.

[0006] In a first aspect, a training method for a fuzzy testing optimization model is provided, comprising:

[0007] A fuzzy test optimization model is established according to preset information; the preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model includes a strategy network and a value network, the strategy network is used to optimize the strategy; the value network is used to calculate the expected return value of the strategy;

[0008] Using a preset algorithm to train the fuzzy test optimization model and determine the optimization strategy;

[0009] Generate test cases according to the optimization strategy;

[0010] Testing the test case in a simulation environment and an actual environment to obtain test feedback data, and adjusting the parameters of the fuzzy test optimization model according to the test feedback data;

[0011] Among them, the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution results of the test case data, and the enhanced state includes: the number of discovered vulnerabilities, code coverage and test case historical information; the preset action space includes the following actions for updating test cases: generating new test cases, intelligently mutating current test cases and merging multiple test cases; the preset strategy includes test case generation rules, and the preset strategy adopts a reinforcement learning structure.

[0012] In a second aspect, a fuzz testing optimization method is provided, comprising:

[0013] Get the target test case;

[0014] Inputting the target test case into a target fuzz test optimization model, so as to perform fuzz testing on the target test case based on a target optimization strategy through the fuzz test optimization model, and obtain a fuzz test result corresponding to the target test case;

[0015] Wherein, the target fuzzy test optimization model and the target optimization strategy are trained by the fuzzy test optimization model training method described in the first aspect above.

[0016] In a third aspect, a training device for a fuzzy testing optimization model is provided, comprising:

[0017] A model building module, used to build a fuzzy test optimization model according to preset information; the preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model includes a strategy network and a value network, the strategy network is used to optimize the strategy; the value network is used to calculate the expected return value of the strategy;

[0018] A training module, used to train the fuzzy test optimization model using a preset algorithm and determine an optimization strategy;

[0019] A test case generation module, used to generate test cases according to the optimization strategy;

[0020] An adjustment module, used to test the test case in a simulation environment and an actual environment to obtain test feedback data, and adjust the parameters of the fuzzy test optimization model according to the test feedback data;

[0021] Among them, the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution results of the test case data, and the enhanced state includes: the number of discovered vulnerabilities, code coverage and test case historical information; the preset action space includes the following actions for updating test cases: generating new test cases, intelligently mutating current test cases and merging multiple test cases; the preset strategy includes test case generation rules, and the preset strategy adopts a reinforcement learning structure.

[0022] In a fourth aspect, a fuzzy test optimization device is provided, comprising:

[0023] A test case acquisition module is used to acquire target test cases;

[0024] A test result acquisition module, used for inputting the target test case into a target fuzzy test optimization model, so as to perform fuzzy testing on the target test case based on a target optimization strategy through the fuzzy test optimization model, and obtain a fuzzy test result corresponding to the target test case;

[0025] Wherein, the target fuzzy test optimization model and the target optimization strategy are trained by the fuzzy test optimization model training method described in the first aspect above.

[0026] In a fifth aspect, an electronic device is provided, the electronic device comprising:

[0027] one or more processors;

[0028] A storage device for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the fuzzy testing optimization model described in the first aspect above, or implement the fuzzy testing optimization method described in the second aspect above.

[0030] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the fuzzy test optimization model described in the first aspect of the embodiment of the present disclosure is implemented, or the fuzzy test optimization method described in the second aspect is implemented.

[0031] In the seventh aspect, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements the training method of the fuzzy testing optimization model described in the first aspect, or implements the fuzzy testing optimization method described in the second aspect.

[0032] The embodiment of the present disclosure provides a training and optimization method, device, equipment and medium for a fuzzy test optimization model, including: establishing a fuzzy test optimization model according to preset information; the preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model includes a strategy network and a value network, the strategy network is used to optimize the strategy; the value network is used to calculate the expected return value of the strategy; the fuzzy test optimization model is trained using a preset algorithm and an optimization strategy is determined; a test case is generated according to the optimization strategy; the test case is tested in a simulation environment and an actual environment to obtain test feedback data, and the parameters of the fuzzy test optimization model are adjusted according to the test feedback data; wherein the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution result of the test case data, and the enhanced state includes: the number of vulnerabilities found, the code coverage and the test case history information; the preset action space includes the following actions for updating the test case: generating a new test case, intelligently mutating the current test case and merging multiple test cases; the preset strategy includes a test case generation rule, and the preset strategy adopts a reinforcement learning structure. This technical solution uses a preset algorithm fuzzy test optimization model for training, determines the optimization strategy, generates test cases according to the optimization strategy, and adjusts the parameters of the fuzzy test optimization model according to the test cases, thereby optimizing the generation of test cases and improving the efficiency and accuracy of fuzzy testing.

[0033] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the embodiments of the present disclosure. Other features of the embodiments of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a flowchart of a training method of a fuzzy test optimization model provided in the first embodiment of the present disclosure;

[0036] Figure 2 is a flow chart of a fuzzy test optimization method provided in the second embodiment of the present disclosure;

[0037] Figure 3 It is a structural schematic diagram of a training device for a fuzzy test optimization model provided in Embodiment 3 of the present disclosure;

[0038] Figure 4 It is a structural diagram of a fuzzy test optimization device provided in Embodiment 4 of the present disclosure;

[0039] Figure 5 It is a structural diagram of an electronic device provided in Embodiment 5 of the present disclosure. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of the present disclosure.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0042] Embodiment 1

[0043] Figure 1 This is a flowchart of a method for training a fuzzy test optimization model provided in the first embodiment of the present disclosure. This embodiment is applicable to the case of training a fuzzy test optimization model. The method can be executed by a training device for a fuzzy test optimization model. The training device for a fuzzy test optimization model can be implemented in the form of hardware and / or software. The training device for a fuzzy test optimization model can be configured in an electronic device. Electronic devices include but are not limited to computers, computers, terminals, servers and other devices with data processing capabilities. Figure 1 As shown, the method includes:

[0044] S110. Establish a fuzzy test optimization model according to preset information; the preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model includes a strategy network and a value network, the strategy network is used to optimize the strategy; the value network is used to calculate the expected return value of the strategy.

[0045] In this embodiment, the fuzzy test process can be modeled as a reinforcement learning process, and a fuzzy test optimization model can be established according to preset information, wherein the fuzzy test optimization model can be a model for optimizing fuzzy testing, the fuzzy test optimization model can be a reinforcement learning model, and the fuzzy test optimization model includes a policy network and a value network, the policy network is used to optimize the policy; the value network is used to calculate the expected return value of the policy. Exemplarily, the policy network can be used to generate a policy or optimize the policy, that is, to select the best action under a given state. The policy network can be a neural network, the input is the state of the environment, the output is the probability distribution corresponding to each possible action, and the goal of the policy network is to maximize the expected cumulative reward. The value network can be used to evaluate the value of a state or a state-action pair, and can be used to predict a given state or the expected return from the state. The value network can help evaluate the quality of different states, so as to make better decisions, and the output of the value network can be a scalar value, indicating the expected return starting from the current state. At the same time, the value network can also output the value of each possible action to help select the best action.

[0046] Continuing with the above description, the preset information may be information used to construct a fuzzy test optimization model, and the preset information may include a preset state space, a preset action space, a preset reward function, and a preset strategy. Among them, the preset state space may be a collection of all states, including a basic state and an enhanced state. The basic state may be a state that can be directly observed, and the basic state may include: test case data and the execution results of the test case data; the enhanced state may be additional information derived from the basic state, and the enhanced state may include: more advanced information such as the number of discovered vulnerabilities, code coverage, and test case history information. The preset action space may be a collection of all possible actions taken in each state, and the preset action space may be used to generate test cases and / or mutate the current test cases. The preset action space includes the following actions for updating test cases: generating new test cases, intelligently mutating current test cases, and merging multiple test cases; the preset reward function may be used to determine the timely reward obtained after executing a specified action, and provide feedback for the fuzzy test optimization model. The preset strategy may be a mapping from state to action, and the action corresponding to the state may be determined according to the state. The preset strategy may adopt a reinforcement learning structure, such as: Q learning or deep Q network (Deep Q-Network, DQN).

[0047] For example, when a new vulnerability is found during the fuzz testing process, the reward function can give a high positive reward; when the code coverage rate increases, the reward function can give a positive reward; when the execution time of the fuzz testing process decreases, the reward function can give an appropriate reward, such as adding 0.1 to the reward value; when invalid tests and / or repeated tests occur during the fuzz testing process, the reward function can give a negative reward. The reward value is adjusted in real time through the reward function, which enhances the adaptability of the fuzz testing optimization model.

[0048] S120, using a preset algorithm to train the fuzzy test optimization model and determine the optimization strategy.

[0049] In this embodiment, the constructed fuzzy test optimization model can be trained using a preset algorithm, and the optimization strategy can be determined according to the policy network during the training process, wherein the optimization strategy can be a strategy generated by the policy network during the training process of the fuzzy test optimization model. The preset algorithm can be a Q-learning algorithm, which can be a simple model-free reinforcement learning algorithm that finds the best strategy for taking a specific action in a given state by learning an action-value function (usually expressed as a Q-value). The Q-value can be used to estimate the expected reward that can be obtained by taking a certain action and moving to the next state.

[0050] S130. Generate test cases according to the optimization strategy.

[0051] It should be explained that after determining the optimization strategy, a test case can be generated according to the optimization strategy. A test case can refer to a set of well-defined test inputs, execution conditions, test steps, and expected results, which are used to verify whether the functions of the software meet its design and requirements. In this embodiment, generating a test case may include generating a new test case, mutating an existing test case, and / or merging multiple test cases.

[0052] Among them, generating new test cases can include two generation methods: random generation and intelligent generation. Among them, random generation can be to generate random input data according to a random generator. Exemplarily, the random input data can include: random strings, random numerical values ​​and / or random Boolean values. Randomly generated test cases are simple and direct, and can help explore possible input boundaries and abnormal situations of the system. Intelligent generation can be based on domain knowledge and rules, and a specific generation algorithm is designed to generate test cases that meet expectations. Intelligent generation of test cases can include generating valid test inputs according to the communication protocol or data format of the industrial control system to cover key functions and boundary conditions of the system.

[0053] Continuing with the above description, mutation of existing test cases may include: mutation based on syntax analysis and semantic mutation. Among them, mutation based on syntax analysis can be a reasonable mutation operation based on the syntax structure and format of the existing test cases. Specifically, it may include modifying data values, adding or deleting part of the data content to generate new test cases while maintaining grammatical correctness. Semantic mutation can be a deeper mutation operation based on the semantic information of the test case, such as the meaning of the input data and the expected results of the operation. For example, for the processing of sensor data, mutation can be performed based on the physical meaning of the data and the working principle of the equipment to simulate various situations that may occur in the real environment.

[0054] Continuing with the above description, merging multiple test cases can be effective information merging, which can be understood as merging the effective parts of multiple test cases to generate new test cases. Specifically, effective information merging can be extracting information fragments from multiple existing effective test cases, and then combining these information fragments into new test cases. Effective information merging can increase the diversity and coverage of test cases, especially when dealing with complex system interactions and data processing logic.

[0055] S140, testing the test cases in a simulation environment and an actual environment to obtain test feedback data, and adjusting the parameters of the fuzzy test optimization model according to the test feedback data.

[0056] Specifically, after obtaining the test case according to the optimization strategy, the test case can be tested in a simulation environment and an actual environment, wherein the simulation environment can be an artificially created environment, usually implemented by a computer program, which can be used to simulate the situation in the real world, and can be used for learning and testing. Testing the test case in the simulation environment can be: through multiple simulation runs, calculating the average vulnerability discovery rate, code coverage, execution time and other indicators. The actual environment can be an environment that can directly interact with the real world, and the testing process of the test case in the actual environment will be affected by actual physical factors. Exemplarily, testing the test case in the actual environment can be: the test case can be applied in an actual industrial control system, its performance and effect in a real environment can be evaluated, and the actual test results can be recorded and analyzed.

[0057] Continuing with the above description, test feedback data can be obtained by testing test cases in a simulated environment and an actual environment. Test feedback data can be feedback data generated during the test process and / or after the test is completed. Exemplarily, the test feedback data may include data such as vulnerability discovery rate, code coverage, execution efficiency and / or strategy convergence. Among them, the vulnerability discovery rate can be the number of vulnerabilities found during the test process; the code coverage rate can be the ratio of the number of lines of code covered by the test case to the total number of lines of code; the execution efficiency can be the number of test cases generated and executed per unit time; and the strategy convergence can be the convergence speed and stability of the strategy during the training process.

[0058] It can be known that after obtaining the test feedback data, the parameters of the fuzzy test optimization model can be adjusted according to the test feedback data. The fuzzy policy optimization model may include a policy network and a value network, and adjusting the parameters of the fuzzy test optimization model may include adjusting the parameters of the policy network and the parameters of the value network, wherein adjusting the parameters of the policy network can be understood as adjusting the optimization strategy. Exemplarily, adjusting the parameters of the fuzzy test optimization model may include: adjusting the learning rate, discount factor and other hyperparameters of the fuzzy test optimization model to improve the learning efficiency of the model; adjusting the parameters of the policy network and the value network to improve the adaptability of the optimization strategy; and adjusting the preset reward function to make the preset reward function more in line with the needs of the actual testing process.

[0059] The present embodiment provides a training method for a fuzzy test optimization model, comprising: establishing a fuzzy test optimization model according to preset information; the preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model includes a strategy network and a value network, the strategy network is used to optimize the strategy; the value network is used to calculate the expected return value of the strategy; the fuzzy test optimization model is trained using a preset algorithm and an optimization strategy is determined; a test case is generated according to the optimization strategy; the test case is tested in a simulation environment and an actual environment to obtain test feedback data, and the parameters of the fuzzy test optimization model are adjusted according to the test feedback data; wherein the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution result of the test case data, and the enhanced state includes: the number of vulnerabilities found, the code coverage rate and the test case history information; the preset action space includes the following actions for updating the test case: generating a new test case, intelligently mutating the current test case and merging multiple test cases; the preset strategy includes a test case generation rule, and the preset strategy adopts a reinforcement learning structure, optimizes the generation of test cases, and improves the accuracy of fuzzy testing.

[0060] As an optional implementation, the preset algorithm in this embodiment may be a Q learning algorithm;

[0061] The method of using a preset algorithm to train the fuzzy test optimization model includes:

[0062] a1) initializing model parameters of the fuzzy test optimization model, wherein the model parameters include action value parameters and strategy network parameters;

[0063] In this embodiment, the fuzzy test optimization model can be trained using a Q learning algorithm. The model parameters of the fuzzy test optimization model can be initialized. Specifically, the parameters of the policy network and the value network can be initialized. In this embodiment, the model parameters may include action value parameters and policy network parameters, wherein the action value parameters may be used to evaluate the expected return of taking a specific action in a specific state. The model parameters may also include weights and biases of the policy network and the value network.

[0064] It should be noted that, when initializing the model parameters of the fuzzy test optimization model, the preset state space, the preset action space and / or the preset strategy may also be set.

[0065] b1) in the current state of the fuzzy test optimization model, selecting a target action in the preset action space according to the preset strategy; the current state belongs to the preset state space;

[0066] In this embodiment, each time may correspond to a state of the fuzzy test optimization model. In the current state of the fuzzy test optimization model, a target action may be selected in a preset action space according to a preset strategy, wherein the current state belongs to the preset state space.

[0067] Exemplarily, when selecting a target action in a preset action space, an ∈-greedy strategy can be used to select the target action, wherein the ∈-greedy strategy can be a basic method for balancing exploration and utilization in reinforcement learning, and a choice can be made between a known best action and trying a new action. Exemplarily, a new action can be chosen to be tried with a probability of ∈, and a known best action can be chosen with a probability of 1-∈.

[0068] c1) executing the target action in a preset environment to obtain an execution result corresponding to the target action; the execution result includes a next state; and the next state belongs to the preset state space.

[0069] Specifically, after the target action is determined, the target action can be executed in a preset environment to obtain an execution result corresponding to the target action, wherein the preset environment can be a pre-set environment, the execution result can be feedback provided by the preset environment after the target action is executed, and the execution result can include the next state. The next state in this embodiment belongs to the preset state space. The next state can be the state of the preset environment after the target action is executed, and can be a specific representation of the preset environment after the target action is executed, which can be understood as the new state of the preset environment after the target action is executed. The execution result can also include some additional information, such as: the degree of proximity to the target, time, resource consumption of the preset environment, and / or whether the action is accepted by the preset environment.

[0070] d1) based on the execution result, obtaining an instant reward according to the preset reward function;

[0071] Specifically, after obtaining the execution result, the instant reward can be calculated based on the execution result using a preset reward function.

[0072] e1) storing the current state, the target action, the next state and the immediate reward as samples in an experience replay pool;

[0073] Specifically, after obtaining the timely reward through calculation, the current state, target action, next state and immediate reward can be stored in the experience replay pool, where the experience replay pool (Experience Replay Pool), also known as the replay buffer, is a technology used in reinforcement learning (deep reinforcement learning), which can be used for the experience gained during the interaction with the preset environment during training (i.e., the combination of the current state, target action, immediate reward and next state) for subsequent training.

[0074] f1) returning to perform the above operations of selecting a target action, executing the target action, obtaining an immediate reward, and storing samples multiple times based on the next state to obtain samples in multiple states;

[0075] It can be known that after the current state, the target action, the next state and the immediate reward are stored as samples in the experience replay pool, the next state is regarded as the current state of the fuzzy test optimization model, and the next state is continued to be executed according to the next state. The target action is selected in the preset action space, the target action is executed, and the immediate reward calculated after the target action is executed is obtained, and the current state (the next state obtained in the last training), the target action, the next state and the immediate reward are stored as samples in the experience replay pool. Through multiple trainings and storing the trained samples, the corresponding samples under multiple states are obtained.

[0076] g1) randomly extracting samples from the experience replay pool for training the fuzzy testing optimization model;

[0077] Specifically, samples can be randomly drawn from the experience replay pool to train the fuzzy testing optimization model, which breaks the correlation between data and improves the training effect of the model.

[0078] h1) End the training according to the preset training end conditions.

[0079] In this embodiment, whether to terminate the training can be determined according to a preset training termination condition, wherein the preset training termination condition can be a pre-set condition for determining the termination of the training. The preset training condition includes at least one of the following: the variation range of the action value parameter is within a preset range, and the strategy network converges; the preset number of training times and / or training time is reached.

[0080] Optionally, before randomly extracting samples from the experience replay pool for training the fuzzy test optimization model, the training method of the fuzzy test optimization model provided in this embodiment further includes:

[0081] a2) for each training, using a Q learning algorithm, updating the action value parameter according to the current state and the target action, the action value parameter being the first expected reward value obtained by following the current strategy and taking the action in the current state;

[0082] In this embodiment, during each training process, the Q learning algorithm can be used to update the action value parameter according to the current state and the target action, wherein the action value parameter can be the first expected reward value obtained by following the current strategy and taking the target action in the current state. It should be explained that different current states and target actions correspond to different first expected reward values, respectively. When the current state and the target action are updated, the first expected reward value can be updated according to the previous first expected reward value. The first expected reward value can be the expected reward of taking the target action in the current state.

[0083] Exemplarily, the first expected return value can be updated based on the Bellman equation, which can be expressed as:

[0084] Q(S t ,A t )←Q(S t ,A t )+α[R t+1 +γmaxQ(S t+1 ,a)-Q(S t ,A t )]

[0085] Among them, t can be the current time, S tCan be the current state, A t Can be a target action, S t+1 It can be the next state, R t+1 Can be the current state from S t Updated to S t+1 The immediate reward corresponding to this process is maxQ(S t+1 ,a) can be the maximum action value parameter corresponding to all target actions in the next state, a can be the target action in the next state, α can be the learning rate, and γ can be the discount factor.

[0086] b2) selecting a new target action in the preset action space according to the action value parameter and the current state and entering the next training.

[0087] It can be known that after the action value parameters are updated according to the current state and the target action, a new target action can be selected in the preset action space according to the updated action value parameters and the current state to enter the next training.

[0088] Optionally, after randomly extracting samples from the experience replay pool for training the fuzzy test optimization model, the training of the fuzzy test optimization model provided in this embodiment further includes:

[0089] a3) for each training, calculating the policy gradient of the second expected return value of the current policy of the training relative to the current parameters of the policy network;

[0090] Specifically, during each training process, the policy gradient of the second expected reward value of the current strategy relative to the current parameters of the strategy network can be calculated, wherein the second expected reward value can be the expected reward value obtained by executing the target action according to the current strategy for a certain current state.

[0091] b3) updating the policy network parameters using a gradient ascent method according to the policy gradient to determine an optimization strategy;

[0092] The second expected return value is the sum of the expected values ​​corresponding to the multiple instant rewards; each expected return value is calculated through the value network.

[0093] Continuing with the above description, after obtaining the policy gradient, the policy network parameters can be updated using the gradient ascent method to determine the optimization strategy, wherein the gradient ascent method is an optimization algorithm that can be used to find the maximum value of a function. In this embodiment, the gradient ascent method can be used to find the parameters corresponding to the policy network with the largest second expected return value.

[0094] Exemplarily, the parameters of the policy network updated using the gradient ascent method can be expressed as:

[0095] θ←θ+α·▽ θ J(θ)

[0096] Where θ can be the parameter of the policy network, J(θ) can be the second expected return, ▽ θ J(θ) may be the policy gradient of the second expected return value of the current policy relative to the current parameters of the policy network, and α may be the learning rate used to control the step size of the parameter update of the policy network.

[0097] The training method of the fuzzy testing optimization model provided in this embodiment first models the fuzzy testing process as a reinforcement learning problem, including a preset state space, a preset action space, a preset reward function and a preset strategy; a reinforcement learning model is established according to the preset state space, the preset action space and the preset reward function; and a suitable reinforcement learning algorithm is selected to obtain the fuzzy testing optimization model and the optimization strategy; test cases are dynamically generated according to the optimized strategy, including random generation, intelligent mutation and merging operations, covering various states and paths of the system; the test cases are tested using the fuzzy testing optimization model and the optimization strategy, the test feedback data is continuously collected, the strategy and parameters are dynamically adjusted, and the test case generation and vulnerability discovery effects are optimized.

[0098] Embodiment 2

[0099] Figure 2 This is a flowchart of a fuzzy test optimization method provided in the second embodiment of the present disclosure. This embodiment is applicable to the case of optimizing fuzzy tests. The method can be executed by a fuzzy test optimization device. The fuzzy test optimization device can be implemented in the form of hardware and / or software. The fuzzy test optimization device can be configured in an electronic device. Figure 2 As shown, the method includes:

[0100] S210: Obtain target test cases.

[0101] In this embodiment, a target test case may be obtained, and the target test case may be a test case to be tested.

[0102] S220, inputting the target test case into the target fuzzy test optimization model, so as to perform fuzzy testing on the target test case based on the target optimization strategy through the fuzzy test optimization model, and obtain the fuzzy test result corresponding to the target test case;

[0103] Among them, the target fuzzy test optimization model and the target optimization strategy are obtained by training the fuzzy test optimization model training method described in any embodiment of the present disclosure.

[0104] In this embodiment, after obtaining the target test case, the target test case can be input into the target fuzzy test optimization model, so as to perform fuzzy testing on the target test case based on the target optimization strategy through the fuzzy test optimization model, and obtain the fuzzy test result corresponding to the target test case. Among them, the target fuzzy test optimization model and the target optimization strategy can be trained by the training method of the fuzzy test optimization model described in any embodiment of the disclosed embodiment.

[0105] The present embodiment provides a fuzz test optimization method, including: obtaining a target test case; inputting the target test case into a target fuzz test optimization model, so as to perform a fuzz test on the target test case based on a target optimization strategy through the fuzz test optimization model, and obtain a fuzz test result corresponding to the target test case; wherein the target fuzz test optimization model and the target optimization strategy are trained by the training method of the fuzz test optimization model described in any embodiment of the present disclosure. The present technical solution uses the trained target fuzz test optimization model and target optimization strategy to test the target test case, thereby improving the accuracy and efficiency of the fuzz test.

[0106] Embodiment 3

[0107] Figure 3 is a structural diagram of a training device for a fuzzy test optimization model provided in Embodiment 3 of the present disclosure; Figure 3 As shown, the device includes: a model building module 310, a training module 320, a test case generation module 330, and an adjustment module 340.

[0108] The model building module 310 is used to build a fuzzy test optimization model according to preset information; the preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; the fuzzy test optimization model includes a strategy network and a value network, the strategy network is used to optimize the strategy; the value network is used to calculate the expected return value of the strategy;

[0109] A training module 320, for training the fuzzy test optimization model using a preset algorithm and determining an optimization strategy;

[0110] A test case generation module 330, used to generate test cases according to the optimization strategy;

[0111] An adjustment module 340, configured to test the test case in a simulation environment and an actual environment to obtain test feedback data, and adjust the parameters of the fuzzy test optimization model according to the test feedback data;

[0112] Among them, the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution results of the test case data, and the enhanced state includes: the number of discovered vulnerabilities, code coverage and test case historical information; the preset action space includes the following actions for updating test cases: generating new test cases, intelligently mutating current test cases and merging multiple test cases; the preset strategy includes test case generation rules, and the preset strategy adopts a reinforcement learning structure.

[0113] Embodiment 3 of the present disclosure provides a training device for a fuzzy test optimization model, which optimizes the generation of test cases and effectively improves the speed and accuracy of fuzzy test optimization.

[0114] Optionally, the preset algorithm is a Q learning algorithm; the training module 320 is used to:

[0115] Initializing model parameters of the fuzzy test optimization model, wherein the model parameters include action value parameters and strategy network parameters;

[0116] In the current state of the fuzzy test optimization model, a target action is selected in the preset action space according to the preset strategy; the current state belongs to the preset state space;

[0117] Executing the target action in a preset environment to obtain an execution result corresponding to the target action; the execution result includes a next state; the next state belongs to the preset state space;

[0118] Based on the execution result, obtaining an instant reward according to the preset reward function;

[0119] The current state, the target action, the next state, and the immediate reward are stored as samples in an experience replay pool;

[0120] Based on the next state, the above operations of selecting a target action, executing the target action, obtaining an immediate reward, and storing the sample are performed multiple times to obtain samples in multiple states;

[0121] Randomly extracting samples from the experience replay pool for training the fuzz testing optimization model;

[0122] End the training according to the preset training end conditions.

[0123] Optionally, the training module 320 is used to:

[0124] For each training, the Q learning algorithm is used to update the action value parameter according to the current state and the target action, and the action value parameter is the first expected reward value obtained by following the current strategy and taking the action in the current state;

[0125] Select a new target action in the preset action space according to the action value parameter and the current state and enter the next training;

[0126] Optionally, the training module 320 is used to:

[0127] For each training, calculating the policy gradient of the second expected return value of the current policy of the training relative to the current parameters of the policy network;

[0128] According to the policy gradient, the policy network parameters are updated using a gradient ascent method to determine an optimization strategy;

[0129] The second expected return value is the sum of the expected values ​​corresponding to the multiple instant rewards; each expected return value is calculated through the value network.

[0130] Optionally, the preset training condition includes at least one of the following:

[0131] The action value parameter changes within a preset range and the strategy network converges;

[0132] Achieve the preset number of training sessions and / or training time.

[0133] The training device for the fuzzy test optimization model provided in the embodiments of the present disclosure can execute the training method for the fuzzy test optimization model provided in any embodiment of the embodiments of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method.

[0134] Embodiment 4

[0135] Figure 4 is a structural diagram of a fuzzy test optimization device provided in the fourth embodiment of the present disclosure; Figure 4 As shown, the device includes: a test case acquisition module 410 and a test result acquisition module 420.

[0136] A test case acquisition module 410 is used to acquire a target test case;

[0137] A test result acquisition module 420 is used to input the target test case into a target fuzzy test optimization model, so as to perform fuzzy testing on the target test case based on a target optimization strategy through the fuzzy test optimization model, and obtain a fuzzy test result corresponding to the target test case;

[0138] Wherein, the target fuzzy test optimization model and the target optimization strategy are obtained by training the fuzzy test optimization model training method described in Example 1.

[0139] Embodiment 4 of the present disclosure provides a fuzzy test optimization device, which improves the accuracy and efficiency of fuzzy testing.

[0140] The fuzzy test optimization device provided in the embodiments of the present disclosure can execute the fuzzy test optimization method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0141] Embodiment 5

[0142] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present disclosure described and / or claimed herein.

[0143] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0144] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0145] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microprocessors, etc. The processor 11 executes the various methods and processes described above, such as the training method of the fuzzy test optimization model and the fuzzy test optimization method.

[0146] In some embodiments, the training method of the fuzzy testing optimization model and the fuzzy testing optimization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the fuzzy testing optimization model and the fuzzy testing optimization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the fuzzy testing optimization model and the fuzzy testing optimization method in any other appropriate manner (e.g., by means of firmware).

[0147] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The computer programs for implementing the methods of the embodiments of the present disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of the disclosed embodiments, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0151] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0152] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions of the embodiments of the present disclosure can be achieved, and this document does not limit this.

[0154] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present disclosure shall be included in the protection scope of the embodiments of the present disclosure.

[0155] The embodiments of the present disclosure also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements a fuzzy test optimization model training method and a fuzzy test optimization method as provided in any embodiment of the present application.

[0156] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operation of the disclosed embodiments, the programming languages ​​including object-oriented programming languages, such as Java, Smalltalk, C++, and also including conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0157] Note that the above are only preferred embodiments of the embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the embodiments of the present disclosure are not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of the present disclosure. Therefore, although the embodiments of the present disclosure are described in more detail through the above embodiments, the embodiments of the present disclosure are not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the embodiments of the present disclosure, and the scope of the embodiments of the present disclosure is determined by the scope of the attached claims.

Claims

1. A training method for a fuzzy testing optimization model, It is characterized in that include: Establish a fuzzy testing optimization model based on preset information; The preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; The fuzzy test optimization model includes a strategy network and a value network, wherein the strategy network is used to optimize the strategy; and the value network is used to calculate the expected return value of the strategy; Using a preset algorithm to train the fuzzy test optimization model and determine the optimization strategy; Generate test cases according to the optimization strategy; Testing the test case in a simulation environment and an actual environment to obtain test feedback data, and adjusting the parameters of the fuzzy test optimization model according to the test feedback data; Among them, the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution results of the test case data, and the enhanced state includes: the number of discovered vulnerabilities, code coverage and test case historical information; the preset action space includes the following actions for updating test cases: generating new test cases, intelligently mutating current test cases and merging multiple test cases; the preset strategy includes test case generation rules, and the preset strategy adopts a reinforcement learning structure.

2. The method according to claim 1, It is characterized in that The preset algorithm is a Q learning algorithm; The method of training the fuzzy test optimization model by using a preset algorithm includes: Initializing model parameters of the fuzzy test optimization model, wherein the model parameters include action value parameters and strategy network parameters; In the current state of the fuzzy test optimization model, a target action is selected in the preset action space according to the preset strategy; the current state belongs to the preset state space; Executing the target action in a preset environment to obtain an execution result corresponding to the target action; the execution result includes a next state; the next state belongs to the preset state space; Based on the execution result, obtaining an instant reward according to the preset reward function; The current state, the target action, the next state, and the immediate reward are stored as samples in an experience replay pool; Based on the next state, the above operations of selecting a target action, executing the target action, obtaining an immediate reward, and storing the sample are performed multiple times to obtain samples in multiple states; Randomly extracting samples from the experience replay pool for training the fuzz testing optimization model; End the training according to the preset training end conditions.

3. The method according to claim 2, It is characterized in that Before randomly extracting samples from the experience replay pool for training the fuzz testing optimization model, the method further includes: For each training, the Q learning algorithm is used to update the action value parameter according to the current state and the target action, and the action value parameter is the first expected reward value obtained by following the current strategy and taking the action in the current state; Select a new target action in the preset action space according to the action value parameter and the current state and enter the next training; After randomly extracting samples from the experience replay pool for training the fuzz testing optimization model, the method further includes: For each training, calculating the policy gradient of the second expected return value of the current policy of the training relative to the current parameters of the policy network; According to the policy gradient, the policy network parameters are updated using a gradient ascent method to determine an optimization strategy; The second expected return value is the sum of the expected values ​​corresponding to the multiple instant rewards; each expected return value is calculated through the value network.

4. The method according to claim 3, It is characterized in that The preset training conditions include at least one of the following: The action value parameter changes within a preset range and the strategy network converges; Achieve the preset number of training sessions and / or training time.

5. A fuzz testing optimization method, It is characterized in that include: Get the target test case; Inputting the target test case into a target fuzz test optimization model, so as to perform fuzz testing on the target test case based on a target optimization strategy through the fuzz test optimization model, and obtain a fuzz test result corresponding to the target test case; Wherein, the target fuzzy test optimization model and the target optimization strategy are obtained by training the fuzzy test optimization model training method described in any one of claims 1-4.

6. A training device for a fuzzy testing optimization model, It is characterized in that include: A model building module is used to build a fuzzy test optimization model according to preset information; The preset information includes a preset state space, a preset action space, a preset reward function and a preset strategy; The fuzzy test optimization model includes a strategy network and a value network, wherein the strategy network is used to optimize the strategy; and the value network is used to calculate the expected return value of the strategy; A training module, used to train the fuzzy test optimization model using a preset algorithm and determine an optimization strategy; A test case generation module, used to generate test cases according to the optimization strategy; An adjustment module, used to test the test case in a simulation environment and an actual environment to obtain test feedback data, and adjust the parameters of the fuzzy test optimization model according to the test feedback data; Among them, the preset state space includes a basic state and an enhanced state, the basic state includes: test case data and the execution results of the test case data, and the enhanced state includes: the number of discovered vulnerabilities, code coverage and test case historical information; the preset action space includes the following actions for updating test cases: generating new test cases, intelligently mutating current test cases and merging multiple test cases; the preset strategy includes test case generation rules, and the preset strategy adopts a reinforcement learning structure.

7. A fuzzy testing optimization device, It is characterized in that include: A test case acquisition module is used to acquire target test cases; A test result acquisition module, used for inputting the target test case into a target fuzzy test optimization model, so as to perform fuzzy testing on the target test case based on a target optimization strategy through the fuzzy test optimization model, and obtain a fuzzy test result corresponding to the target test case; Wherein, the target fuzzy test optimization model and the target optimization strategy are obtained by training the fuzzy test optimization model training method described in any one of claims 1-4.

8. An electronic device, It is characterized in that include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the fuzzy testing optimization model as described in any one of claims 1-4, or implement the fuzzy testing optimization method as described in claim 5.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, it implements the training method of the fuzzy test optimization model as described in any one of claims 1 to 4, or implements the fuzzy test optimization method as described in claim 5.

10. A computer program product, It is characterized in that The computer program product includes a computer program, which, when executed by a processor, implements the training method of the fuzzy testing optimization model as described in any one of claims 1 to 4, or implements the fuzzy testing optimization method as described in claim 5.

Citation Information

Cited By

  • Application test method and device, electronic equipment, product and storage medium

    CN119938531A

  • Intelligent electromechanical equipment test management method and system, electronic equipment and storage medium

    CN120296544A

  • Intelligent parameter generation method combining combinatorial optimization algorithm and fuzzy test

    CN120492359A