Intelligent contract vulnerability detection method and system based on deep reinforcement learning

Abstract

The invention provides an intelligent contract vulnerability detection method based on deep reinforcement learning. The intelligent contract vulnerability detection method comprises the steps that a scheduler selects an action according to a strategy in tth training; the action is applied to the target smart contract, and a new fuzzy test trial case is generated and executed; collecting an execution result after the tth training; according to an execution result, calculating a total reward and a state of the round of training, and generating and storing an experience playback tetrad; sampling batch data to update parameters of a scheduler; repeating the steps until a preset target is reached, and obtaining an optimized scheduler; and performing fuzzy testing on the new smart contract by adopting the optimized scheduler to obtain vulnerability information and a test case. According to the method, exploratory performance is guaranteed, meanwhile, usability is improved, and the efficiency and accuracy of vulnerability detection are remarkably improved. The invention further provides an intelligent contract vulnerability detection system based on deep reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of smart contract security detection technology, specifically to a smart contract vulnerability detection method and system based on deep reinforcement learning. Background Technology

[0002] As a core component of blockchain technology, smart contracts have been widely applied in key areas such as decentralized finance, supply chain management, and digital asset trading. Their immutable nature once deployed on a blockchain network means that any potential security vulnerability could cause irreparable and enormous economic losses.

[0003] Currently, smart contract vulnerability detection methods are mainly divided into two categories: static analysis and dynamic analysis. Static analysis methods can detect some known vulnerabilities without running the contract code, but they have inherent drawbacks such as high false positive rates, prominent path explosion problems, and difficulties in solving complex constraints. Fuzzing, a dynamic analysis method, effectively discovers unknown vulnerabilities and path-related defects by inputting a large amount of mutated test data into the contract and monitoring its runtime behavior, thus becoming a research hotspot in this field.

[0004] However, existing smart contract fuzzing techniques often rely on random selection for mutation strategies, failing to adaptively adjust mutation operators based on real-time feedback during testing. This leads to underutilization of high-value mutation operations while inefficient mutations are frequently used, resulting in severely uneven distribution of testing resources and low vulnerability discovery efficiency. Furthermore, the lack of or overly simplistic test case scheduling mechanisms leads to high redundancy in generated test cases, wasting significant computational resources on exploring similar or repetitive execution paths, making it difficult to uncover critical, deep vulnerability triggering conditions. Moreover, existing methods generally fail to fully consider the unique semantic characteristics of the Ethereum Virtual Machine, such as gas consumption mechanisms, storage models, and cross-contract call logic, causing a disconnect between the testing process and the underlying runtime environment, making it difficult to effectively trigger high-risk vulnerabilities in real-world scenarios.

[0005] The above-mentioned problems urgently need to be solved. Summary of the Invention

[0006] Based on the aforementioned problems in the existing technology, the purpose of this invention is to provide a smart contract vulnerability detection method and system based on deep reinforcement learning. By analyzing the execution feedback information collected during the fuzzing process in real time, the system dynamically selects high-value mutation operators and combines them with a lightweight probabilistic scheduling strategy. This approach enhances exploitability while ensuring exploratory capabilities, thereby significantly improving the efficiency and accuracy of vulnerability detection.

[0007] The technical solution adopted by this invention to solve its technical problem is: a smart contract vulnerability detection method based on deep reinforcement learning, comprising:

[0008] S1, a scheduler built on a DQN network, selects action a according to the ∈-greedy policy in the t-th training iteration. t ;

[0009] S2, action a t The corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzz test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if the vulnerability is not triggered, the execution information is collected for training updates.

[0010] S3 collects the execution results after the t-th training, including vulnerability trigger information, code coverage, critical opcode information, data path information, comparison path information, function call sequence, and 8 types of EVM opcode information;

[0011] S4. Calculate the corresponding reward value based on the vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information. Then, perform a weighted fusion calculation to determine the total reward R for the t-th action. t ;

[0012] S5, based on the function call sequence and 8 types of EVM opcode information, obtain the state S of this training round. t And the state S of the (t-1)th training session t-1 Action a t Reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,a t ,R t ,S t And store it in the experience replay pool;

[0013] S6, sample batch data from the experience replay pool and update the scheduler parameters in combination with the priority replay strategy;

[0014] S7. Repeat the above steps until the preset number of training rounds or test duration is reached, thereby obtaining the optimized scheduler.

[0015] S8 uses an optimized scheduler to perform fuzz testing on new smart contracts, dynamically selects the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, runs test cases, and obtains vulnerability information and test cases through multiple rounds of testing.

[0016] Furthermore, in S1, the scheduler built based on the DQN network selects action a according to the ∈-greedy policy in the t-th training iteration. t include:

[0017] Step S11, the scheduler is composed of a DQN network;

[0018] Step S12, the action a t To select the mutation operator to apply to fuzz testing;

[0019] Step S13, the ∈-greedy strategy is to use the scheduler's output as the action a for this round with probability ∈. t With probability 1-∈, randomly select an action from all action groups as the action a to be used in this round. t This is to ensure exploration in unknown spaces.

[0020] Furthermore, in S2, the action a... t The corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzz test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if the vulnerability is not triggered, execution information is collected for training updates, including:

[0021] Step S21, the mutation objects are the ABI parameters and EVM parameters of the target smart contract;

[0022] Step S22, perform action a t One of the corresponding mutation operators is applied to the mutation object, including: converting the numerical value of the target mutation object into a byte stream representation; randomly selecting one or more byte units in the byte stream as mutation targets; generating new byte values ​​according to a uniform or Gaussian distribution; and recombining the mutated ABI parameters and EVM parameters according to the contract call format to form a complete fuzzy test case x'.

[0023] Step S23: Run fuzz test case x' in EVM and record the execution results during the execution process; if a vulnerability is triggered during the execution of the fuzz test case, the training process of the sample ends and the next training is directly started; if no vulnerability is triggered, proceed to the next step and continue to collect execution information for training updates, so as to ensure that the training process can continuously explore new execution paths under different combinations of mutated objects and mutation operators.

[0024] Furthermore, the execution results are collected using the log module configured in the experiment.

[0025] The log module includes a vulnerability detection module, a coverage monitoring module, an opcode tracking module, a storage access tracking module, an instruction execution tracking module, and a function call tracking module;

[0026] The vulnerability detection module is used to collect vulnerability triggering information;

[0027] The coverage monitoring module is used to collect code coverage information;

[0028] The opcode tracking module is used to collect key opcode information and 8 types of EVM opcode information;

[0029] The storage access tracking module is used to collect data path information;

[0030] The instruction execution tracking module is used to collect and compare path information;

[0031] The function call tracing module is used to collect function call sequence information.

[0032] Furthermore, in S4, the reward R for the t-th action is calculated using a reward function based on vulnerability triggering information, code coverage, critical opcode information, data path information, and comparison path information. t include:

[0033] Step S41: Calculate the reward value corresponding to the vulnerability trigger information;

[0034] The formula for calculating the reward value corresponding to the vulnerability triggering information is as follows:

[0035] R vul =w vul *I vul

[0036] Among them, R vul The reward value corresponding to the vulnerability trigger information, I vul =1 indicates that the current operation triggered the vulnerability, and 0 indicates that it was not triggered; w vul Weights are assigned to vulnerability types and can be adjusted based on vulnerability severity.

[0037] Step S42: Calculate the reward value corresponding to the code coverage information;

[0038] The formula for calculating the reward value corresponding to the code coverage information is as follows:

[0039] R cov =w branch *ΔB+w inst *ΔI

[0040] Among them, R cov Here, ΔB represents the reward value corresponding to code coverage information, ΔI represents the number of branches newly covered, and w represents the number of instructions newly covered. branch The weights corresponding to the number of newly covered branches, w inst The weights correspond to the number of newly covered instructions;

[0041] Step S43: Calculate the reward value corresponding to the key operation code information;

[0042] The reward calculation formula corresponding to the key operation code information is as follows:

[0043] R inst =∑w k *I k for k = 1 to 8

[0044] Among them, R inst w is the reward value corresponding to the key opcode information. k This is the opcode weight, where k is the total number of critical opcodes; I k Boolean indicator variable I k If this opcode appears in the trajectory, then I k =1, otherwise I k =0;

[0045] Step S44: Calculate the reward value corresponding to the data path information.

[0046] The formula for calculating the reward value corresponding to the data path information is as follows:

[0047] R datawp =I datawp

[0048] Among them, R datawp The reward value corresponding to the data path information, I datawp =1 indicates that the condition is met, and 0 indicates that the condition is not met;

[0049] Step S45: Calculate the reward value corresponding to the comparison path information;

[0050] The reward calculation formula corresponding to the comparison path information is as follows:

[0051] R comwp =I comwp

[0052] Among them, R comwp To compare the reward values ​​corresponding to the path information, I comwp =1 indicates that the current operand is less than the historical minimum, triggering a reward; if the condition is not met, then it is 0.

[0053] Step S46: Weight and fuse the reward values ​​corresponding to vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information to obtain the final total reward R. t ;

[0054] The total reward R t The calculation formula is:

[0055] R t =w vul *R vul +wcov *R cov +w inst *R inst +w datawp *R datawp +w comwp *R comwp

[0056] Among them, R t For the total reward, w vul For vulnerability-triggered reward weighting, R vul w is the reward value corresponding to the vulnerability trigger information. cov R assigns weights to code coverage rewards cov w is the reward value corresponding to code coverage information. inst As a reward weight for critical opcodes, R inst w is the reward value corresponding to the key opcode information. datawp R is the data path reward weight. datawp w is the reward value corresponding to the data path information. comwp To compare path reward weights, R comwp To compare the reward values ​​corresponding to the path information.

[0057] Furthermore, in S5, the state S of this training round is obtained based on the function call sequence and the 8 types of EVM opcode information. t And the state S of the (t-1)th training session t-1 Action a t Total reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,a t ,R t ,S t And storing it in the experience replay pool includes:

[0058] Step S51: Encode the call order, call depth, and call parameter combination in the function call sequence to form the function call sequence state vector S. func ;

[0059] Step S52: Arrange the relative frequencies of the 8 types of EVM opcode information in a fixed order to form the opcode frequency state vector S. op ;

[0060] Step S53, convert the function call sequence state vector S func and opcode frequency state vector S op The weighted combination yields the state S for this round of training. t ;

[0061] The state S of this round of trainingt The weighted formula is:

[0062]

[0063] Among them, S t Based on the current training status, S func w is the state vector of the function call sequence. func S represents the weights corresponding to the state vector of the function call sequence. op w is the opcode frequency state vector. op The weights corresponding to the opcode frequency state vector. This is a vector concatenation operation;

[0064] Step S54, construct the experience replay quadruple {S t-1 ,a t ,R t ,S t}, including the state S during the (t-1)th training iteration. t-1 Action a t Total reward R t and the state S during the t-th training session t .

[0065] Furthermore, in S6, the step of sampling batch data from the experience replay pool and updating the scheduler's parameters in conjunction with the priority replay strategy includes:

[0066] Step S61: Sample a batch of data from the experience replay pool based on the priority of time difference error;

[0067] The time difference error formula is as follows:

[0068]

[0069] Where, δ t Q(S) represents the time difference error, and γ is the discount factor; t ,a′;θ - R represents the predictive value of the current DQN network for this state-action pair. t For the total reward, θ represents the maximum expected value of the target network for the next state; θ represents the current network parameters. - For the updated network parameters;

[0070] Step S62: Calculate the target Q value and construct the loss function based on the mean square error. Calculate the loss gradient using the backpropagation algorithm and iteratively update the DQN network parameters using the Adam optimizer.

[0071] The formula for calculating the target Q value is:

[0072] Q target=R t +γmax a′ Q(S t ,a′;θ - )

[0073] Among them, Q target For the target Q value, R t Let γ be the total reward, θ be the discount factor, and θ be the total reward. - For the updated network parameters, max a′ Q(S t ,a′;θ - ) is the target network for the next state S t The maximum value of the action.

[0074] This invention also provides a smart contract vulnerability detection system based on deep reinforcement learning, the system comprising:

[0075] The action selection module is used in the scheduler built on the DQN network. The scheduler selects action a according to the ∈-greedy policy in the t-th training iteration. t ;

[0076] The test case generation and execution module is used to generate action a. t The corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzz test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if the vulnerability is not triggered, the execution information is collected for training updates.

[0077] The execution result collection module is used to collect the execution results after the t-th training. The execution results include vulnerability trigger information, code coverage, critical opcode information, data path information, comparison path information, function call sequence, and 8 types of EVM opcode information.

[0078] The comprehensive reward calculation module is used to calculate the corresponding reward value based on vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information, and then perform a weighted fusion calculation to calculate the total reward R for the t-th action. t ;

[0079] The state update module is used to obtain the current training state S based on the function call sequence and 8 types of EVM opcode information. t ;

[0080] The experience storage module is used to store the state S of the (t-1)th training iteration. t-1 Action a t Reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,at ,R t ,S t And store it in the experience replay pool;

[0081] The parameter update module is used to sample batch data from the experience replay pool and update the scheduler's parameters in conjunction with the priority replay strategy.

[0082] The iterative training convergence module is used to repeat the above steps until the preset number of training rounds or test duration is reached, thereby obtaining the optimized scheduler.

[0083] The vulnerability detection application module is used to perform fuzz testing on new smart contracts using an optimized scheduler. It dynamically selects the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, runs test cases, and obtains vulnerability information and test cases through multiple rounds of testing.

[0084] The present invention also provides a computer-readable storage medium storing one or more instructions, wherein the one or more instructions execute the above-described method for detecting smart contract vulnerabilities based on deep reinforcement learning.

[0085] The present invention also provides an electronic device, comprising: a memory and a processor; the memory storing at least one program instruction; the processor loading and executing the at least one program instruction to implement the above-described method for detecting smart contract vulnerabilities based on deep reinforcement learning.

[0086] The beneficial effects of this invention are as follows: The smart contract vulnerability detection method based on deep reinforcement learning of this invention, by introducing a deep reinforcement learning scheduler, can dynamically select high-value mutation operators based on real-time feedback information, avoiding inefficient random mutation and redundant seed problems, and improving the accuracy and efficiency of vulnerability detection while ensuring exploratory nature; by introducing a noise layer and priority experience replay, it can maintain stable detection performance in large-scale real contracts, has good cross-contract generalization ability, and is suitable for application in complex and diverse blockchain ecosystems; by using reinforcement learning algorithms to automatically optimize the scheduling strategy, it significantly reduces the dependence on manual experience and rule design, reduces manual tuning costs, and improves the automation and intelligence level of the testing process; it improves the detection rate and accuracy in typical vulnerabilities and real contracts, can quickly locate multiple types of security risks, and provides strong protection for the secure operation of blockchain smart contracts. Attached Figure Description

[0087] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0088] Figure 1 This is a flowchart of the smart contract vulnerability detection method based on deep reinforcement learning provided in Embodiment 1 of the present invention.

[0089] Figure 2 This is a schematic diagram of the smart contract vulnerability detection system based on deep reinforcement learning provided in Embodiment 2 of the present invention.

[0090] Figure 3 This is a partial block diagram of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0091] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0092] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0093] First implementation method:

[0094] The first embodiment of the present invention provides a smart contract vulnerability detection method based on deep reinforcement learning, comprising: a scheduler constructed based on a DQN network, wherein the scheduler selects action a according to the ∈-greedy policy in the t-th training iteration. t ; to perform action a t The corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzz test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if no vulnerability is triggered, execution information is collected for training updates; the execution results after the t-th training are collected; and the total reward R is calculated based on the execution results. t and the state of this round of training S tThe process involves generating experience replay quadruples and storing them in an experience replay pool; sampling batch data from the experience replay pool and updating the scheduler parameters using a priority replay strategy; repeating the above steps until a preset number of training rounds or testing duration is reached, resulting in an optimized scheduler; using the optimized scheduler to perform fuzz testing on new smart contracts, dynamically selecting the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, running test cases, and obtaining vulnerability information and test cases through multiple rounds of testing. This invention's deep reinforcement learning-based smart contract vulnerability detection method dynamically selects high-value mutation operators by analyzing execution feedback information collected during fuzz testing in real time, and combining this with a lightweight probabilistic scheduling strategy. This ensures exploratory nature while improving exploitability, thereby significantly improving the efficiency and accuracy of vulnerability detection. It can adapt to different contract logic structures and operating environments, quickly locating potential risk points in various types of smart contracts, providing an efficient, intelligent, and scalable solution for the security testing of blockchain smart contracts.

[0095] The following details the implementation of the deep reinforcement learning-based smart contract vulnerability detection method in this embodiment. The following content is for ease of understanding and is not essential for implementing this solution. The specific process of this embodiment is as follows: Figure 1 As shown, this implementation method is applied to a smart contract vulnerability detection system based on deep reinforcement learning.

[0096] Step S1: A scheduler built based on a DQN network selects action a according to the ∈-greedy policy during the t-th training iteration. t .

[0097] Specifically, the scheduler built on the DQN network selects action a according to the ∈-greedy policy in the t-th training iteration. t Includes the following steps:

[0098] Step S11, the scheduler is composed of a DQN network.

[0099] Specifically, the main structure of the DQN network consists of an input layer, hidden layers, and an output layer. The hidden layers employ a multi-layer fully connected neural network, with each layer containing several neurons. The activation function used is ReLU, and dropout or batch normalization is introduced to enhance the network's generalization ability and training stability.

[0100] To enhance exploration capabilities on unknown contract structures, this invention introduces a Noisy mechanism into the DQN network. This involves introducing learnable noise terms after some fully connected layers in the hidden layers, incorporating trainable noise parameters into the weights and biases to achieve randomness in action selection. Specifically, each weight and bias in the network is represented as: θ = μ + σ⊙∈, where μ is a learnable mean parameter, σ is a learnable standard deviation parameter, ∈ represents noise sampled from a zero-mean distribution (typically a standard normal distribution or a factorized Gaussian distribution), and ⊙ denotes element-wise multiplication. In this way, the noise is learnable, allowing the network to automatically adjust its exploration amplitude instead of relying on a fixed ∈-greedy strategy. Compared to traditional fixed ∈-greedy strategies, the DQN network with the Noisy mechanism maintains a high level of exploration in the early stages of training, discovering more unknown paths and potential vulnerabilities. In the later stages of training, the noise gradually decreases, causing the strategy to converge to the optimal action selection, thus significantly improving the efficiency and effectiveness of the intelligent scheduler in fuzz testing.

[0101] The DQN network takes the smart contract execution state generated in the subsequent step S5 as its input and outputs the optimal mutation operator selection probability for the current state.

[0102] Step S12, the action a t To select the mutation operator to apply to fuzz testing.

[0103] Specifically, there are 17 mutation operators, including ByteRandMutator (mutates bytes by random assignment), ByteInterestingMutator (replaces bytes with boundary or special values), DwordAddMutator (performs additive perturbation on double-word data), ByteNegMutator (negates byte values), ByteIncMutator (increments byte values), ByteAddMutator (performs addition on byte values), ByteFlipMutator (flips byte bits), ByteDecMutator (decreases byte values), and BytesSwapMutator (swaps byte sequences). The following are some of the byte arrays: BitFlipMutator (flips bits), ConstantHintedMutator (replaces a value with a preset constant or hint value), BytesRandSetMutator (randomly sets a value at a specific position in a byte sequence), WordInterestingMutator (replaces a word or half-word with a boundary value or special value), DwordInterestingMutator (replaces double-word data with a boundary value or special value), BytesSetMutator (directly sets a byte sequence to a specific value), WordAddMutator (performs addition on words or half-words), and QwordAddMutator (performs addition on quad-word data).

[0104] Step S13, the ∈-greedy strategy is to use the scheduler's output as the action a for this round with probability ∈. t With probability 1-∈, randomly select an action from all action groups as the action a to be used in this round. t This is to ensure exploration in unknown spaces.

[0105] This mechanism can prevent the scheduler from getting stuck in local optima, thus maintaining the ability to explore new vulnerability triggering conditions.

[0106] Step S2, perform action a t The corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzz test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if the vulnerability is not triggered, the execution information is collected for training updates.

[0107] Specifically, the action a tThe corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzz test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if the vulnerability is not triggered, execution information is collected for training updates, including the following steps:

[0108] Step S21, the mutated objects are the ABI parameters and EVM parameters of the target smart contract.

[0109] Specifically, EVM (Ethereum Environment) parameters include EVM account balance, EVM block gas limit, gas price, block number, base fee, call value, block timestamp, random number seed, block address, and difficulty.

[0110] The ABI parameters of a smart contract include: ABI empty parameter (TEmpty), static type parameter (TStatic, T256), dynamic type parameter (TDynamic), array type parameter (TArray), and unknown type parameter (TUnknown).

[0111] Step S22, perform action a t One of the corresponding mutation operators is applied to the mutation object, including: converting the numerical value of the target mutation object into a byte stream representation; randomly selecting one or more byte units in the byte stream as mutation targets; generating new byte values ​​according to a uniform or Gaussian distribution; and recombining the mutated ABI parameters and EVM parameters according to the contract call format to form a complete fuzzy test case x'.

[0112] As an example, the selected action a t The corresponding mutation operator is ByteRandMutator (which mutates bytes by randomly assigning values), and the mutation target is the EVM account balance. The specific mutation process is as follows: First, the numerical value of the EVM account balance is encoded as a byte stream representation; then, one or more byte units in the byte stream are randomly selected as mutation targets; next, new byte values ​​are generated according to a uniform or Gaussian distribution and replace the original values ​​of the selected target byte; furthermore, this process can be repeated at multiple positions in the balance byte stream to form multi-point random mutation. Through the above mutation method, a large number of different account balance input values ​​can be generated without relying on semantic constraints, thereby effectively increasing the diversity of test inputs, expanding the coverage of smart contract execution paths, and increasing the probability of fuzz testing triggering potential vulnerabilities.

[0113] Step S23: Run fuzz test case x' in EVM (Ethereum environment) and record the execution results during the execution process; if a vulnerability is triggered during the execution of the above fuzz test case, the training process of the sample ends and the next training is directly started; if no vulnerability is triggered, proceed to the next step and continue to collect execution information for training updates, so as to ensure that the training process can continuously explore new execution paths under different combinations of mutated objects and mutated operators.

[0114] Step S3: Collect the execution results after the t-th training. The execution results include vulnerability trigger information, code coverage, critical opcode information, data path information, comparison path information, function call sequence, and 8 types of EVM opcode information.

[0115] Specifically, the execution results are collected using the log module configured in the experiment. The log module includes a vulnerability detection module, a coverage monitoring module, an opcode tracing module, a storage access tracing module, an instruction execution tracing module, and a function call tracing module.

[0116] Among them, the vulnerability detection module is used to collect vulnerability trigger information; the coverage monitoring module is used to collect code coverage information; the opcode tracing module is used to collect key opcode information and 8 types of EVM opcode information; the storage access tracing module is used to collect data path information; the instruction execution tracing module is used to collect comparison path information; and the function call tracing module is used to collect function call sequence information.

[0117] Step S4: Calculate the corresponding reward values ​​based on the vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information. Then, perform a weighted fusion calculation to determine the total reward R for the t-th action. t .

[0118] Specifically, the reward R for the t-th action is calculated using a reward function based on vulnerability triggering information, code coverage, critical opcode information, data path information, and comparison path information. t Includes the following steps:

[0119] Step S41: Calculate the reward value corresponding to the vulnerability trigger information.

[0120] Based on execution traces, exception return information, event logs, and other data, the data is transmitted to the vulnerability detection module. The vulnerability detection module identifies the type of vulnerability trigger based on predefined vulnerability detection rules and symbolic execution results. If a vulnerability is detected, a fixed reward is given to the scheduler.

[0121] The reward value is weighted according to the importance of the vulnerability type. The formula for calculating the reward value for vulnerability trigger information is as follows:

[0122] R vul =w vul *I vul

[0123] Among them, R vul The reward value corresponding to the vulnerability trigger information, I vul =1 indicates that the current operation triggered the vulnerability, and 0 indicates that it was not triggered; w vul This is the weight for vulnerability types, which can be adjusted according to the severity of the vulnerability.

[0124] The vulnerability types are weighted as follows: Reentrancy vulnerabilities have a significant impact on contract fund security, with a weight of 1.0; Integer overflow / underflow vulnerabilities may lead to abnormal asset transfers, with a weight of 0.7; Assertion failure vulnerabilities typically manifest as failed assert or require statements, and if they lead to fund locking, they are of medium severity, with a weight of 0.5; Unchecked call return value vulnerabilities may allow for logic bypassing, with a weight of 0.6; Timestamp dependency vulnerabilities may be manipulated by verifiers, with a weight of 0.4; Insecure randomness vulnerabilities affect the fairness of lottery-type contracts, with a weight of 0.5; Denial of Service vulnerabilities (with Unexpected Revert / Block GasLimit) affect availability, with a weight of 0.6; Access Control / Authorization Flaw vulnerabilities may lead to arbitrary calls to sensitive functions, posing a significant threat, with a weight of 0.9; Short Address attacks... Attacks, under certain circumstances, cause parameter truncation and have a weight of 0.4; Business Logic Bugs, which violate contract business rules, have a weight of 0.8 if they involve fund transfers; Storage Collision / Overwrite Vulnerabilities may lead to data tampering and have a weight of 0.7; Unrestricted Selfdestruct Vulnerabilities can directly destroy the contract and have a weight of 0.9.

[0125] Step S42: Calculate the reward value corresponding to the code coverage information.

[0126] To encourage fuzz testing to explore new execution paths in each round, the number of newly covered code branches and executed instructions is recorded and rewarded. The statistics for code branches and executed instructions are obtained from the coverage monitoring module, which records the cumulative number of covered branches and instructions during the contract virtual machine's execution and updates it after each round. The incremental difference is calculated by comparing the total coverage after the current round with the total coverage of the previous round. The reward value is proportional to the number of newly covered branches and instructions. The formula for calculating the reward value corresponding to code coverage information is as follows:

[0127] R cov =w branch *ΔB+w inst *ΔI

[0128] Among them, R cov Here, ΔB represents the reward value corresponding to code coverage information, ΔI represents the number of branches newly covered, and w represents the number of instructions newly covered. branch The weights corresponding to the number of newly covered branches, w inst The weights correspond to the number of newly covered instructions.

[0129] As an example, experimental verification shows that the optimal weight combination is w. branch =0.6, w inst =0.4, which can both highlight the incentive for branch exploration and ensure that the instruction-level coverage is effectively compensated.

[0130] Step S43: Calculate the reward value corresponding to the key operation code information.

[0131] During smart contract execution, EVM opcodes include SHA3, CALL, CREATE, SELFDESTRUCT, JUMP, JUMPI, SLOAD, and SSTORE. Among these, SSTORE, CALL, and SELFDESTRUCT opcodes are often directly related to high-risk behaviors such as state modification, external calls, or contract destruction. Therefore, they need to be recorded separately and their rewards calculated to guide fuzz testing towards exploring these potentially dangerous paths. During smart contract execution, the opcode tracking module dynamically monitors each EVM instruction running in the Ethereum runtime environment and records the instruction sequence in the execution trajectory log. Subsequently, the system scans this trajectory to detect whether it contains a predefined set of critical opcodes. For each opcode, a Boolean indicator variable I is set. k If this opcode appears in the trajectory, then I k =1, otherwise I k =0.

[0132] The formula for calculating the reward value corresponding to the key operation code information is as follows:

[0133] R inst =∑w k *I k for k = 1 to 8

[0134] Among them, R inst w is the reward value corresponding to the key opcode information. k Let k be the weight of the opcode, and k be the total number of critical opcodes.

[0135] Weights are assigned to eight categories of critical opcodes: SELFDESTRUCT directly destroys the contract, posing an extremely high risk, with a weight of 1.0; CALL performs external calls, potentially triggering reentrancy attacks, with a weight of 0.9; CREATE dynamically generates a new contract, potentially triggering logic bloat or attack surface expansion, with a weight of 0.8; SSTORE modifies the contract's storage state, affecting contract logic and fund security, with a weight of 0.5; SLOAD reads stored data, typically a critical branch condition in the contract logic, with a weight of 0.4; JUMPI conditional jump determines the execution path branch, with a weight of 0.6; JUMP unconditional jump may increase control flow complexity, with a weight of 0.3; SHA3 calculates the storage location hash or is used for random logic, potentially affecting storage indexes and security, with a weight of 0.2.

[0136] Through this hierarchical weighting design, fuzzing, driven by the rewards of deep reinforcement learning, prioritizes exploring paths involving high-risk behaviors (such as contract destruction, external calls, and dynamic creation), while also considering the coverage of conditional branches and storage operations. This mechanism can significantly improve the probability of triggering potential vulnerabilities and the comprehensiveness of vulnerability detection.

[0137] Step S44: Calculate the reward value corresponding to the data path information.

[0138] To guide the scheduler to focus on potential vulnerabilities related to storage data consistency (such as logical inconsistencies and state tampering), a storage access tracking module is introduced during smart contract execution to monitor storage read and write behavior in real time. The storage access tracking module records each SSTORE and SLOAD instruction, including the storage slot location and its corresponding value. Based on these records and the execution path, the system determines whether the new value t written to the storage slot is different from the historical value t-1, indicating that the write operation has indeed changed the content of the storage slot; and whether the storage slot was previously read in the execution path.

[0139] When both of the above conditions are met, the system marks the execution point as a data path point of interest and awards a reward. The formula for calculating the reward value corresponding to the data path information is as follows:

[0140] R datawp =I datawp

[0141] Among them, R datawp The reward value corresponding to the data path information, I datawp =1 indicates that the condition is met, and 0 indicates that the condition is not met.

[0142] Through this mechanism, the scheduler can more effectively explore execution paths related to storage consistency, thereby improving its ability to detect logical and state-related vulnerabilities.

[0143] Step S45: Calculate the reward value corresponding to the comparison path information.

[0144] During smart contract execution, comparison instructions (such as LT, GT, EQ, ISZERO, etc.) typically influence the branch selection of the control flow. Fuzzing, if it can continuously approach the critical values ​​of these branch conditions, is more likely to explore new execution paths or trigger boundary conditions. Therefore, this system incorporates comparison path information into its reward mechanism. To achieve this, an instruction execution tracking module is set up during execution, which captures all comparison instructions and their operands. Here, the current operand refers to the two values ​​actually pushed onto the stack and involved in the comparison instruction's calculation during contract runtime (e.g., storage slot value, function call parameters, environment variable msg.value, block.timestamp, etc.).

[0145] During the comparison, the system defines the difference between the two operands as:

[0146] distance=|operand1-operand2|

[0147] Where distance is the difference between the two operands, and operand1 and operand2 represent the two input parameters of the current comparison instruction.

[0148] These values ​​can come from contract storage, function input parameters, blockchain runtime environment parameters, or intermediate computation results. The system maintains a historical minimum distance table for the location of each comparison instruction, recording the minimum difference encountered in all previous executions at that location. If the operand difference obtained from the current execution is less than the historical minimum, it indicates that the fuzzing has explored that branch condition more deeply. In this case, the system identifies the comparison point as a new comparison path point of interest and awards a reward.

[0149] The reward calculation formula corresponding to the comparison path information is as follows:

[0150] R comwp =I comwp

[0151] Among them, Rcomwp To compare the reward values ​​corresponding to the path information, I comwp =1 indicates that the current operand is less than the historical minimum, triggering a reward; if the condition is not met, it is 0.

[0152] Through this mechanism, the scheduler can gradually "approach" the critical values ​​of key branch conditions in smart contracts, thereby effectively improving path coverage and increasing the likelihood of triggering boundary-related vulnerabilities (such as integer overflow, condition bypass, state inconsistency, etc.).

[0153] Step S46: Weight and fuse the reward values ​​corresponding to vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information to obtain the final total reward R. t .

[0154] Specifically, to comprehensively leverage the guiding role of different types of execution information in the fuzzing exploration process, a reward aggregation module is set up to calculate the total reward value after each round of execution. This module weights and merges the reward values ​​corresponding to vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information to obtain the final total reward R. t Total reward R t The calculation formula is:

[0155] R t =w vul *R vul +w cov *R cov +w inst *R inst +w datawp *R datawp +w comwp *R comwp

[0156] Among them, R t For the total reward, w vul The reward weight for triggering the vulnerability is set at 0.4; R vul w is the reward value corresponding to the vulnerability trigger information. cov The code coverage reward weight is set to 0.25; R cov w is the reward value corresponding to code coverage information. inst The critical opcode reward weight is set to 0.15; R inst w is the reward value corresponding to the key opcode information. datawp The data path reward weight is set to 0.10; R datawp w is the reward value corresponding to the data path information. comwp To determine the path reward weight for comparison, a value of 0.10 is used; R comwp To compare the reward values ​​corresponding to the path information.

[0157] The weight parameters can be set to the recommended values ​​mentioned above, or they can be dynamically adjusted according to different testing objectives. For example, in testing scenarios that emphasize path diversity, the weight can be increased. cov The proportion; and in scenarios where the focus is on detecting logical consistency vulnerabilities, it can increase w vul The proportion.

[0158] Step S5: Obtain the current training state S based on the function call sequence and the 8 types of EVM opcode information. t And the state S of the (t-1)th training session t-1 Action a t Reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,a t ,R t ,S t And store it in the experience replay pool.

[0159] Specifically, the state S of this training round is obtained based on the function call sequence and 8 types of EVM opcode information. t And the state S of the (t-1)th training session t-1 Action a t Total reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,a t ,R t ,S t} and storing it in the experience replay pool includes the following steps:

[0160] Step S51: Encode the call order, call depth, and call parameter combination in the function call sequence to form the function call sequence state vector S. func .

[0161] Specifically, to characterize the function call features during the execution of smart contracts, the system includes a function call tracing module to record the complete function call chain and generate a function call sequence. The function call sequence includes the call order (the sequence in which each function is triggered), the call depth (the function's position in the call stack), and the call parameters (the parameter values ​​input during function execution).

[0162] The function call tracing module encodes the call order, call depth, and call parameters to form a function call sequence state vector. This vector reflects the call relationships between functions and their operational characteristics, providing support for subsequent state modeling and reward calculation.

[0163] Step S52: Arrange the relative frequencies of the 8 types of EVM opcode information in a fixed order to form the opcode frequency state vector S. op .

[0164] Specifically, to characterize the instruction distribution features during contract execution, an opcode tracking module is used to capture and statistically analyze the usage of eight predefined key EVM opcodes. The opcode tracking module first counts the occurrence of each type of opcode during contract execution. Then, the relative frequency of each type of opcode is defined as the ratio of its occurrence count to the total occurrence count of all eight types of opcodes. Finally, the relative frequencies of the eight types of opcodes are arranged in a fixed order to obtain the opcode frequency state vector.

[0165] The eight types of EVM opcodes include: SHA3, CALL, CREATE, SELFDESTRUCT, JUMP, JUMPI, SLOAD, and SSTORE.

[0166] Opcode frequency state vector S op It can form a fixed-dimensional state vector that reflects the distribution characteristics of different types of opcodes during contract execution, providing a quantitative basis for vulnerability detection and path exploration.

[0167] Step S53, convert the function call sequence state vector S func and opcode frequency state vector S op The weighted combination yields the state S for this round of training. t .

[0168] Specifically, in order to comprehensively characterize the feature information during the execution process of smart contracts, a state aggregation module is set up to calculate the state S of the current training round after each round of execution. t The state aggregation module will generate the function call sequence state vector S. func With opcode frequency state vector S op We perform a weighted combination to obtain the state S of this training round. t The state of S in this round of training t The weighted formula is:

[0169]

[0170] Among them, S t Based on the current training status, S func w is the state vector of the function call sequence. func The weights corresponding to the state vectors of the function call sequence are set to 0.6; S op w is the opcode frequency state vector. op is the weight corresponding to the opcode frequency state vector, which is set to 0.4; ⊕ is the vector concatenation operation.

[0171] Weights can be configured according to testing requirements and analysis objectives to flexibly highlight call relationships or underlying instruction characteristics in the state representation. The state aggregation module can provide the scheduler with differentiated state representations while maintaining information integrity, enabling it to obtain effective state input under different testing objectives, thereby improving the accuracy and flexibility of scheduling decisions.

[0172] Step S54, construct the experience replay quadruple {S t-1 ,a t ,R t ,S t}, including the state S during the (t-1)th training iteration. t-1 Action a t Total reward R t and the state S during the t-th training session t .

[0173] Specifically, the total reward R for completing each round of smart contract execution t and state S t Then, through the experience replay management module, the state S of the (t-1)th training session is recorded. t-1 Currently executing action a t Current round reward R t And the current training status S t Combined to form a quadruple {S t-1 ,a t ,R t ,S t The samples are then stored in the experience replay pool. When the experience replay pool reaches its capacity limit, its contents are updated according to a first-in-first-out (FIFO) or priority sampling strategy to ensure the freshness and diversity of training samples. The quadruplets stored in the experience replay pool can be used to batch train a deep reinforcement learning scheduler. Through continuous sampling and iterative optimization, the scheduler can gradually learn strategies for efficiently triggering vulnerabilities, improving code coverage, and discovering critical execution paths, thereby continuously optimizing the fuzzing process and improving vulnerability detection efficiency and depth.

[0174] Step S6: Sample batch data from the experience replay pool and update the scheduler parameters in conjunction with the priority replay strategy.

[0175] Specifically, the step of sampling batch data from the experience replay pool and updating the scheduler parameters in conjunction with the priority replay strategy includes the following steps:

[0176] Step S61: Sample a batch of data from the experience playback pool based on the priority of time difference error.

[0177] Specifically, to improve training efficiency and the learning effect of key samples, the system introduces a prioritized experience replay strategy during the sampling process. The system prioritizes experience replay based on each quadruple {S}. t-1 ,a t ,R t ,S t The time difference error (TD) is prioritized to give higher sampling probabilities to samples that have a greater impact on policy network updates, and samples are sampled in batches according to priority. The time difference error formula is:

[0178]

[0179] Where, δ t Q(S) represents the time difference error, and γ is the discount factor; t ,a′;θ - R represents the predictive value of the current DQN network for this state-action pair. t For the total reward, θ represents the maximum expected value of the target network for the next state; θ represents the current network parameters. - These are the updated network parameters.

[0180] The discount factor γ is used to balance the effects of immediate rewards and future rewards.

[0181] Step S62: Calculate the target Q value and construct the loss function based on the mean square error. Calculate the loss gradient using the backpropagation algorithm and iteratively update the DQN network parameters through the Adam optimizer.

[0182] Specifically, during each training round, the system samples training data from the experience replay pool through the experience replay management module to update the scheduler network parameters. The maximum capacity of the experience replay pool is 1000 quadruples {S}. t-1 ,a t ,R t ,S t When the number of samples stored in the pool reaches its limit, newly generated quadruples will replace the oldest samples according to a first-in, first-out (FIFO) strategy to ensure the freshness and diversity of training samples. The system samples a batch of quadruples from the experience replay pool according to priority and calculates the target Q value.

[0183] The formula for calculating the target Q value is:

[0184] Q target =R t +γmax a′ Q(S t ,a′;θ - )

[0185] Among them, Qtarget For the target Q value, R t Let γ be the total reward, θ be the discount factor, and θ be the total reward. - For the updated network parameters, max a′ Q(S t ,a′;θ - ) is the target network for the next state S t The maximum value of the action.

[0186] Constructing a loss function based on mean squared error (MSE) Subsequently, the loss gradient is calculated using the backpropagation algorithm, and the DQN network parameters are iteratively updated using the optimizer Adam. This enables the scheduler to learn progressively to select the optimal action under different states, thereby improving the efficiency of fuzz test case generation and vulnerability triggering capabilities.

[0187] Simultaneously, importance sampling weights are calculated for each sample to correct for biases introduced by non-uniform sampling. During the update process, the system uses a weighted loss function. The DQN network parameters are updated iteratively using backpropagation and the Adam optimizer. After the update, the system readjusts the priority of the quadruples according to the new TD error, and prioritizes the retention of high-priority samples and eliminates low-priority or oldest samples when the experience replay pool capacity reaches its limit. This ensures that the training samples are both fresh and valuable, enabling the scheduler to learn critical state-action pairs efficiently and improve the performance of fuzz testing on critical paths and potential vulnerability triggering.

[0188] Step S7: Repeat the above steps until the preset number of training rounds or test duration is reached, thereby obtaining the optimized scheduler.

[0189] Specifically, steps S1-S6 are repeated, allowing the scheduler to continuously optimize its mutation operator selection strategy through interactive learning. In each training round, the scheduler utilizes and updates historical samples through an experience replay mechanism, and strengthens the learning of high-value samples by combining a priority replay strategy, thereby gradually improving the accuracy and generalization ability of the decision-making process. When the number of training iterations reaches the preset training round limit, or the training runtime exceeds the preset testing duration limit, the training process stops, resulting in a converged optimized scheduler. This optimized scheduler possesses adaptive mutation scheduling capabilities, enabling it to automatically select the optimal mutation operator based on contract execution characteristics during subsequent fuzzing testing, thereby improving the efficiency and coverage of vulnerability detection.

[0190] As an example, the preset training epochs are set to 10,000. The preset testing duration is set to five hours.

[0191] Step S8: Use the optimized scheduler to perform fuzz testing on the new smart contract, dynamically select the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, run test cases, and obtain vulnerability information and test cases through multiple rounds of testing.

[0192] Specifically, the process of using an optimized scheduler to perform fuzz testing on new smart contracts, dynamically selecting the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, running test cases, and obtaining vulnerability information and test cases through multiple rounds of testing includes the following steps:

[0193] Step S81: Initialize and deploy the smart contract code to be tested and the trained and optimized scheduler, and prepare the ABI parameters and Ethereum runtime environment (EVM) parameters required for contract calls as initial data for test input.

[0194] Step S82: Input the ABI interface parameters and Ethereum runtime environment parameters of the smart contract under test into the optimized scheduler; the scheduler will output the optimal action a* based on the initial state and map it to the corresponding mutation operator; use the selected mutation operator to mutate the ABI parameters and EVM parameters of the target smart contract to generate new input data, and combine them to form a complete fuzz test case. x ′.

[0195] Step S83: Run test cases in the Ethereum runtime environment (EVM). x The system collects runtime logs during execution and transmits the log data to the vulnerability detection module. If a vulnerability is triggered, the vulnerability information and corresponding test cases are output and stored in the vulnerability information database. If no vulnerability is triggered, the scheduler selects a new mutation operator to generate and execute new test cases until a vulnerability is triggered or the preset test count limit or time limit is reached. Finally, vulnerability information and its corresponding fuzzy test case set are generated for smart contract security verification and vulnerability localization.

[0196] The first embodiment of the present invention provides a smart contract vulnerability detection method based on deep reinforcement learning, comprising: a scheduler constructed based on a DQN network, wherein the scheduler selects action a according to the ∈-greedy policy in the t-th training iteration. t ; to perform action a tThe corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzzy test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if no vulnerability is triggered, execution information is collected for training updates. The execution results after the t-th training are collected, including vulnerability trigger information, code coverage, critical opcode information, data path information, comparison path information, function call sequence, and 8 types of EVM opcode information. The corresponding reward values ​​are calculated based on the vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information, and then weighted and fused to calculate the total reward R for the t-th action. t The state S of this training round is obtained based on the function call sequence and 8 types of EVM opcode information. t And the state S of the (t-1)th training session t-1 Action a t Reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,a t ,R t ,S tThe data is then stored in the experience replay pool; batch data is sampled from the experience replay pool, and the scheduler parameters are updated in conjunction with the priority replay strategy; the above steps are repeated until the preset training rounds or test duration are reached, thereby obtaining the optimized scheduler; the optimized scheduler is used to perform fuzz testing on the new smart contract, dynamically selecting the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, running test cases, and obtaining vulnerability information and test cases through multiple rounds of testing. This invention's deep reinforcement learning-based smart contract vulnerability detection method introduces a deep reinforcement learning scheduler, enabling dynamic selection of high-value mutation operators based on real-time feedback. This avoids inefficient random mutation and redundant seed problems, improving the accuracy and efficiency of vulnerability detection while maintaining exploratory capabilities. It fully considers Ethereum Virtual Machine (EVM) semantics, Gas constraints, and cross-contract interaction logic; the scheduling mechanism can adaptively adjust according to different contract characteristics, effectively triggering key vulnerability conditions. Employing a structured action space design and a dual-strategy fusion mechanism, it can flexibly expand new mutation operators or test feedback indicators to adapt to different blockchain platforms and application scenarios. By introducing a noise layer and priority experience replay, it maintains stable detection performance in large-scale real contracts, possessing good cross-contract generalization capabilities, making it suitable for application in complex and diverse blockchain ecosystems. Utilizing reinforcement learning algorithms to automatically optimize the scheduling strategy significantly reduces reliance on manual experience and rule design, lowers manual tuning costs, and improves the automation and intelligence of the testing process. It improves the detection rate and accuracy in typical vulnerabilities and real contracts, enabling rapid identification of multiple types of security risks and providing strong protection for the secure operation of blockchain smart contracts.

[0197] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0198] Example 1:

[0199] This study verifies the efficiency and accuracy of the proposed deep reinforcement learning-based smart contract vulnerability detection method in detecting typical vulnerabilities. The experiment selected 130 smart contracts containing typical vulnerabilities from the authoritative VeriSmart-benchmarks and SmartBugs test suites. These contracts were manually annotated, and the vulnerability types covered a variety of common vulnerabilities, including suicide vulnerabilities, Ethereum leaks, and assertion failures. The contract sizes ranged from 5 lines of code to 658 lines, with an average of approximately 116 lines. To objectively evaluate FuzzMaster's performance, the experiment compared it with the mainstream symbolic execution tool SmartTest and three smart contract fuzzing tools: ConFuzzius, Smartian, and Ityfuzz. The experiment used detection accuracy, detection rate, and vulnerability detection time as evaluation metrics. The operating environment was an Ubuntu system equipped with an Intel Xeon Gold 6226 24-core 48-thread CPU, an NVIDIA 2080Ti graphics card, and 125GB of RAM. All tools are tested for 30 minutes on each contract. The timeout threshold for the Z3 solver in SmartTest is set to 90 seconds, while FuzzMaster uses 2-fold cross-validation for training and testing. Each test is repeated 5 times and the average value is taken.

[0200] The experimental results are shown in Table 1 below. FuzzMaster significantly outperforms other tools in both accuracy and detection rate. Specifically, the symbolic execution tool SmartTest only detected approximately 23.08% of vulnerabilities, with an accuracy of 96.77%; the traditional fuzzing tools ConFuzzius and Ityfuzz achieved detection rates of 43.08% and 46.92%, respectively, with accuracies of 98.25% and 92.42%. In contrast, FuzzMaster's policy network (Ptable) version achieved a detection rate of 53.85% and 100% accuracy; while the deep reinforcement learning (DQN) version further improved the detection rate to 66.15%, maintaining 100% accuracy. Furthermore, FuzzMaster demonstrates superior vulnerability triggering speed and exploration efficiency, quickly focusing on vulnerability-sensitive paths, avoiding redundant exploration, and significantly improving testing efficiency. The experimental results fully demonstrate that the deep reinforcement learning-based smart contract fuzzing method proposed in this invention can effectively improve vulnerability detection capabilities and accuracy, outperforming existing symbolic execution and fuzzing tools, and possesses high practical application value.

[0201] Table 1

[0202]

[0203] Example 2:

[0204] This embodiment of the deep reinforcement learning-based smart contract vulnerability detection method focuses on exploring the impact of the noise layer and Priority Experience Replay (PER) module in FuzzMaster on vulnerability detection performance, and verifies the generalization ability of its trained model on real-world deployed smart contracts. The experiment still uses the aforementioned 130 contracts containing typical vulnerabilities as the training set (D1), and further collects 7533 real smart contracts from Etherscan in November 2024 (D2). These contracts, after deduplication and preprocessing, range in size from 6 to 1953 lines of code, with an average of 286 lines. For the core module design, three tool variants are constructed: FuzzMaster Noisy with only a noise layer, FuzzMaster PER with the PER module but without a noise layer, and the complete FuzzMaster standard version including both. Experimental metrics include vulnerability detection time, instruction coverage, and branch coverage. The experiments are conducted in the same hardware environment, and all test results are averaged after multiple repetitions.

[0205] The experimental results are shown in Table 2 below. The addition of the noise layer improved the diversity of test exploration and the vulnerability triggering capability, but it also brought significant detection time consumption and fluctuations, with an average detection time of approximately 60 seconds. The instruction coverage and branch coverage were 65.4% and 59.5%, respectively. The PER module, through its priority sampling mechanism, reduced invalid exploration, stabilizing the detection time at 25 seconds, and also improving coverage (71.1% for instructions and 63.7% for branches). The complete FuzzMaster standard version, combining the advantages of the noise layer and the PER module, achieved optimal performance, further compressing the detection time to 15 seconds, with instruction and branch coverage reaching 75.2% and 68.4%, respectively. The test results on the real contract dataset D2 are even more convincing. FuzzMaster detected 128 vulnerabilities, covering 5 different vulnerability categories, far exceeding ConFuzzius's 23 and Ityfuzz's 53, while the symbolic execution tool SmartTest did not detect any vulnerabilities. This indicates that FuzzMaster not only has good testing efficiency and coverage but can also generalize well to complex and diverse real-world smart contract environments.

[0206] Table 2

[0207]

[0208] In summary, this example verifies the rationality of the FuzzMaster module design. The noise layer and the PER module each play important roles, and their combination yields the best results. Through application on large-scale real-world contracts, it demonstrates that the deep reinforcement learning-based smart contract fuzzing method has strong generalization ability and practical application value, effectively uncovering multiple types of vulnerabilities and ensuring the security and stability of smart contracts.

[0209] Second implementation method:

[0210] Please see Figure 2 The second embodiment of the present invention provides a smart contract vulnerability detection system based on deep reinforcement learning, comprising: an action selection module 201, a test case generation and execution module 202, an execution result collection module 203, a comprehensive reward calculation module 204, a state update module 205, an experience storage module 206, a parameter update module 207, an iterative training convergence module 208, and a vulnerability detection application module 209.

[0211] Specifically, the action selection module 201 is used for a scheduler built on a DQN network, which selects action a according to the ∈-greedy policy in the t-th training iteration. t Test case generation and execution module 202 is used to generate action a t The corresponding mutation operator is applied to the ABI and EVM parameters of the target smart contract to generate new fuzzy test cases and execute them. If the execution triggers a vulnerability, the current sample training process ends and the next training begins; if no vulnerability is triggered, execution information is collected for training updates. The execution result collection module 203 collects the execution results after the t-th training, including vulnerability trigger information, code coverage, critical opcode information, data path information, comparison path information, function call sequence, and 8 types of EVM opcode information. The comprehensive reward calculation module 204 calculates the corresponding reward values ​​based on the vulnerability trigger information, code coverage, critical opcode information, data path information, and comparison path information, and then performs weighted fusion to calculate the total reward r for the t-th action. t The state update module 205 is used to obtain the current training state S based on the function call sequence and 8 types of EVM opcode information. t Experience storage module 206 is used to store the state S of the (t-1)th training iteration. t-1 Action a t Reward R t and the state S during the t-th training session t As an experience replay quadruple {S t-1 ,a t ,R t ,S tThe data is then stored in the experience replay pool; the parameter update module 207 is used to sample batch data from the experience replay pool and update the scheduler parameters in combination with the priority replay strategy; the iterative training convergence module 208 is used to repeat the above steps until the preset training rounds or test duration are reached, thereby obtaining the optimized scheduler; the vulnerability detection application module 209 is used to perform fuzz testing on new smart contracts using the optimized scheduler, dynamically select the optimal mutation operator to mutate the ABI and EVM parameters of the target smart contract, run test cases, and obtain vulnerability information and test cases through multiple rounds of testing.

[0212] It is not difficult to see that this embodiment is a system implementation corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0213] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0214] Third implementation method:

[0215] The third embodiment of the present invention provides a network-side server, such as... Figure 3 As shown, it includes at least one processor 301; and a memory 302 communicatively connected to at least one processor 301; wherein the memory 302 stores commands that can be executed by at least one processor 301, and the instructions are executed by at least one processor 301 to enable at least one processor 301 to perform the above-described data processing method.

[0216] The memory 302 and processor 301 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 301 and memory 302 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 301 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 301.

[0217] Processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 302 can be used to store data used by processor 301 during operation.

[0218] Fourth implementation method:

[0219] A fourth embodiment of the present invention provides a computer read storage medium storing a computer program. When the computer program is executed by a processor, it implements the smart contract vulnerability detection method based on deep reinforcement learning in the first embodiment.

[0220] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0221] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

[0222] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting vulnerabilities of smart contracts based on deep reinforcement learning, characterized in that, The method comprises: S1, a scheduler based on the DQN network, the scheduler determines the action of the agent in the t-th training according to the Q value of each action in the state of the agent in the t-th training policy selection action ; S2, the action The corresponding mutation operator is applied to the ABI parameters and EVM parameters of the target smart contract to generate a new fuzz test test case and execute it. If the execution triggers a vulnerability, the current sample training process is ended and the next training is entered. If no vulnerability is triggered, the execution information is collected for training update. S3, collecting the execution result after the t-th training, wherein the execution result comprises vulnerability trigger information, code coverage, key operation code information, data path information, comparison path information, function call sequence and 8 types of EVM operation code information; S4, according to the vulnerability trigger information, code coverage, key operation code information, data path information and comparison path information, the corresponding reward value is calculated respectively, and then the total reward of the tth action is calculated by weighted fusion ; S5, obtaining the state of the current round of training according to the function call sequence and the 8-class EVM opcode information and the state of the (t-1)th training , action , reward and the state of the tth training as an experience replay quadruple and storing to the experience replay pool; S6, sampling batch data from the experience replay pool, and updating the parameters of the scheduler in combination with the priority replay strategy; S7, repeating the above steps until a preset training round or test duration is reached, thereby obtaining an optimized scheduler; S8, using the optimized scheduler to perform fuzz testing on a new smart contract, dynamically selecting an optimal mutation operator to mutate the ABI parameters and EVM parameters of the target smart contract, running test cases, and obtaining vulnerability information and test cases through multiple rounds of testing.

2. The deep reinforcement learning based intelligent contract vulnerability detection method of claim 1, wherein, In S1, the scheduler constructed based on the DQN network, in the tth training, according to the state of the tth training and the Q value of the state of the tth training, the scheduler selects an action as the policy selection action. Policy selection action Comprise: Step S11, the scheduler is composed of a DQN network; Step S12, the action to select a mutation operator to apply to the fuzzing test; Step S13, the strategy is to select an action from the action set with probability , using the output of the scheduler as the action used this round ; with probability , randomly select an action from all action groups as the action used this round to ensure exploration in the unknown space.

3. The deep reinforcement learning based intelligent contract vulnerability detection method of claim 2, wherein, In S2, the action of The corresponding mutation operator is applied to the ABI parameters and EVM parameters of the target smart contract to generate a new fuzz test test case and execute it. If the execution triggers a vulnerability, the current sample training process is ended, and the next training is entered. If no vulnerability is triggered, the execution information is collected for training and updating, comprising: Step S21, the mutation object is the ABI parameters and EVM parameters of the target smart contract; Step S22, the action The corresponding one of the mutation operators is applied to the mutation object, including: converting the numerical value of the target mutation object into a byte stream representation; randomly selecting one or more byte units in the byte stream as the mutation target; generating a new byte value according to a uniform distribution or a Gaussian distribution; and recombining the mutated ABI parameter and the EVM parameter according to the contract calling format to form a complete fuzz test case ; Step S23, running the fuzzing test case in the EVM record the execution result in the execution process; if a vulnerability is triggered in the execution process of the fuzzing test case, end the training process of the sample and directly enter the next training; if no vulnerability is triggered, enter the subsequent step and continue to collect the execution information for training update, so as to ensure that the training process can continuously explore new execution paths under different combinations of variation objects and variation operators.

4. The deep reinforcement learning based intelligent contract vulnerability detection method of claim 1, wherein, The execution result is collected by a log module set up in the experiment; The log module comprises a vulnerability detection module, a coverage monitoring module, an operation code tracking module, a storage access tracking module, an instruction execution tracking module and a function call tracking module; The vulnerability detection module is used to collect vulnerability trigger information; The coverage monitoring module is used to collect code coverage information; The operation code tracking module is used to collect key operation code information and 8 types of EVM operation code information; The storage access tracking module is used to collect data path information; The instruction execution tracking module is used to collect comparison path information; The function call tracking module is used to collect function call sequence information.

5. The deep reinforcement learning based intelligent contract vulnerability detection method of claim 1, wherein, In S4, the reward of the tth action is calculated by a reward function according to the vulnerability trigger information, the code coverage, the key operation code information, the data path information and the comparison path information comprises: Step S41, calculating the reward value corresponding to the vulnerability trigger information; The reward value corresponding to the vulnerability trigger information is calculated according to the following formula: wherein, is a reward value corresponding to the vulnerability trigger information, =1 indicates that the current operation triggers the vulnerability, and 0 indicates that the vulnerability is not triggered; is a vulnerability type weight, which can be adjusted according to the vulnerability severity; Step S42, calculating the reward value corresponding to the code coverage information; The reward value corresponding to the code coverage information is calculated according to the following formula: wherein, is a reward value corresponding to the code coverage information, is a number of newly covered branches, is a number of newly covered instructions, is a weight corresponding to the number of newly covered branches, is a weight corresponding to the number of newly covered instructions; Step S43, calculating the reward value corresponding to the key operation code information; The reward value corresponding to the key operation code information is calculated according to the following formula: in, The reward value corresponding to the key operation code information. This is the weight of the opcode, where k is the total number of critical opcodes; Boolean indicator variable If this opcode appears in the trajectory, then =1, otherwise =0; Step S44, calculating the reward value corresponding to the data path information The reward value corresponding to the data path information is calculated according to the following formula: wherein, is a reward value corresponding to the data path information, =1 indicates that the condition is met, and 0 indicates that the condition is not met; Step S45, calculating the reward value corresponding to the comparison path information; The reward value corresponding to the comparison path information is calculated according to the following formula: wherein, is a reward value corresponding to the path information, = 1 indicates that the current operation distance is less than the historical minimum value, triggering the reward; otherwise, it is 0. Step S46, weighting and fusing the reward values corresponding to the vulnerability trigger information, the code coverage, the key operation code information, the data path information and the comparison path information to obtain a final total reward ; the total award The calculation formula is: wherein, is a total reward, is a vulnerability trigger reward weight, is a reward value corresponding to the vulnerability trigger information, is a code coverage reward weight, is a reward value corresponding to the code coverage information, is a critical opcode reward weight, is a reward value corresponding to the critical opcode information, is a data path reward weight, is a reward value corresponding to the data path information, is a comparison path reward weight, is a reward value corresponding to the comparison path information.

6. The deep reinforcement learning based intelligent contract vulnerability detection method of claim 4, wherein, In S5, the state of the current round of training is obtained according to the function call sequence and the 8-class EVM opcode information , the state of the (t-1)th training , the action , the total reward , and the state of the tth training are taken as an experience replay four-tuple and stored into an experience replay pool, comprising: Step S51, encode the calling order, calling depth and calling parameter combination in the function call sequence to form a function call sequence state vector ; Step S52, arrange the relative frequency of 8 categories of EVM operation code information in fixed order, form operation code frequency state vector ; Step S53, the function call sequence state vector and the opcode frequency state vector are combined to obtain the state of this round of training ; The state of the present round of training The weighting formula is: wherein, is the state of the current round of training, is the function call sequence state vector, is the function call sequence state vector corresponding weight, is the opcode frequency state vector, is the opcode frequency state vector corresponding weight, and is the vector concatenation operation. Step S54, constructing experience replay quadruple including state of t-1th training , action , total reward and state of tth training .

7. The deep reinforcement learning based intelligent contract vulnerability detection method of claim 6, wherein, In S6, the sampling of batch data from the experience replay pool and the updating of the parameters of the scheduler in combination with the priority replay strategy comprise: Step S61, sampling a batch of data from the experience replay pool based on the priority of the time difference error; The time difference error formula is: wherein, is the time-difference error, is the discount factor; is the predicted value of the state-action pair by the current DQN network; is the total reward, is the maximum expected value of the next state by the target network; is the current network parameter, is the updated network parameter; Step S62, calculating the target Q value and constructing a loss function based on the mean square error, calculating the loss gradient by using the back propagation algorithm, and updating the DQN network parameters through the optimizer Adam iteration; The target Q value calculation formula is: wherein, is the target Q-value, is the total reward, is the discount factor, are the updated network parameters, is the target network's maximum action value for the next state . 8.A deep reinforcement learning based smart contract vulnerability detection system, characterized in that, The system is applied to the smart contract vulnerability detection method based on deep reinforcement learning according to any one of claims 1-7, and the system comprises: An action selection module is configured to select an action based on a DQN network constructed by a scheduler, wherein the scheduler is configured to determine a target value of a state-action pair in the t-th training according to a policy selection action ; The test case generation execution module is configured to execute the action Corresponding mutation operators are applied to the ABI parameters and EVM parameters of the target smart contract to generate new fuzzy test cases and execute them. If the execution triggers a vulnerability, the current sample training process is ended, and the next training is entered. If no vulnerability is triggered, the execution information is collected for training update. An execution result collection module is configured to collect an execution result after the t-th training, wherein the execution result includes vulnerability trigger information, code coverage, key operation code information, data path information, comparison path information, function call sequence, and eight types of EVM operation code information. The comprehensive reward calculation module is configured to calculate corresponding reward values according to the vulnerability trigger information, the code coverage, the key operation code information, the data path information and the comparison path information, and then perform weighted fusion calculation on a total reward of the tth action ; a state updating module, configured to obtain a state of the current round of training according to the function call sequence and the 8-class EVM opcode information ; The experience storage module is used to store the state of the (t-1)th training iteration. ,action ,award and the state of the t-th training session As an experience replay quadruple And store it in the experience replay pool; A parameter updating module is configured to sample batch data from an experience replay pool and update parameters of the scheduler in combination with a priority replay strategy. An iterative training convergence module is configured to repeat the above steps until a preset training round or test duration is reached, thereby obtaining an optimized scheduler. A vulnerability detection application module is configured to perform fuzz testing on a new smart contract by using the optimized scheduler, dynamically select an optimal mutation operator to mutate ABI parameters and EVM parameters of the target smart contract, run test cases, and obtain vulnerability information and test cases through multiple rounds of testing.

9. A computer-readable storage medium having stored therein one or more instructions, wherein The one or more instructions implement the deep reinforcement learning-based smart contract vulnerability detection method of any one of claims 1-7 when executed.

10. An electronic device, comprising: Comprise: A memory and a processor; at least one program instruction is stored in the memory; The processor loads and executes the at least one program instruction to implement the deep reinforcement learning-based smart contract vulnerability detection method of any one of claims 1-7.

Images