Smart contract vulnerability detection method, device and related equipment based on reinforcement learning

Abstract

The present application discloses a method, apparatus, and related equipment for detecting vulnerabilities in smart contracts based on reinforcement learning. The method comprises: constructing an interactive interface between a transaction agent model and a simulated trading platform based on a selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage; connecting the trained transaction agent model to the simulated trading platform through the interactive interface, the transaction agent model determining the function cluster selected in the transaction, the simulated trading platform responding to the selected function cluster and generating corresponding log information; analyzing the log information to obtain vulnerability detection results for the target smart contract; wherein the transaction agent model is obtained through unsupervised learning training using a reinforcement learning algorithm, which can improve the efficiency of fuzz testing and explore deeper execution paths.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and more specifically, to a smart contract vulnerability detection method, apparatus, and related equipment based on reinforcement learning. Background Art

[0002] Currently, blockchain and smart contracts have been applied in a variety of fields, including financial services, public services, and the Internet of Things. The assets managed by smart contracts on various smart contract application platforms have reached trillions of yuan. In the production environment of smart contract application platforms, once deployed, smart contracts cannot be modified. Therefore, discovering vulnerabilities in smart contracts before deployment and mitigating or even eliminating them has become a hot topic among scholars.

[0003] Fuzz testing is a mainstream method for detecting vulnerabilities in smart contracts. Compared to traditional static analysis methods, fuzz testing is faster, more effective, and easier to implement. However, existing fuzz testing tools can easily detect shallow vulnerabilities but struggle to explore deeper paths, making some vulnerabilities in deeper paths difficult to detect. Summary of the Invention

[0004] In view of this, the present application provides a smart contract vulnerability detection method, apparatus and related equipment based on reinforcement learning to detect vulnerabilities in smart contracts.

[0005] To achieve the above objectives, the first aspect of this application provides a smart contract vulnerability detection method based on reinforcement learning, comprising:

[0006] Based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage, an interactive interface between the transaction agent model and the simulated trading platform is constructed;

[0007] Connecting the trained transaction agent model to the simulated transaction platform through the interactive interface, the transaction agent model determining the function cluster selected in the transaction, and the simulated transaction platform responding to the selected function cluster and generating corresponding log information;

[0008] Analyze the log information to obtain vulnerability detection results of the target smart contract;

[0009] The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm.

[0010] Preferably, the interaction interface includes a state interface, an action selection interface and a reward feedback interface;

[0011] The process of constructing an interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the operation code coverage includes:

[0012] Based on the current state of the smart contract and multiple preset function clusters, a transaction agent model and an action selection interface for the simulated trading platform are constructed;

[0013] According to the status of the smart contract, build the status interface between the transaction agent model and the simulated trading platform;

[0014] Based on the growth rate of the opcode coverage, a reward feedback interface between the transaction agent model and the simulated trading platform is constructed.

[0015] Preferably, the action selection interface is an action input from the transaction agent model to the simulated trading platform; the process of constructing the action selection interface between the transaction agent model and the simulated trading platform based on the current state of the smart contract and multiple preset function clusters includes:

[0016] Based on the current state of the smart contract, The greedy algorithm determines the target function cluster from the preset multiple function clusters, and determines the target function from the target function cluster. ;

[0017] Randomly generate objective function The parameter x, the transaction initiator sender and the transaction amount amount are used to construct the action input t:

[0018]

[0019] Among them, the preset multiple function clusters are obtained by clustering the functions contained in different smart contracts through the K-means algorithm in advance, and each function cluster contains several functions with similar functions.

[0020] Preferably, the state interface is the state input from the simulated trading platform to the transaction agent model; the process of constructing the state interface between the transaction agent model and the simulated trading platform according to the state of the smart contract includes:

[0021] Get the state of the smart contract, including the frequency of executed opcodes, the balance of the smart contract, and the frequency of each function cluster being selected;

[0022] The state is converted into a feature vector, and the feature vector is used as the state input from the simulated trading platform to the trading agent model.

[0023] Preferably, the reward feedback interface is a reward input from the simulated trading platform to the trading agent model. The process of constructing the reward feedback interface between the trading agent model and the simulated trading platform based on the growth rate of the opcode coverage rate includes:

[0024] Determine a reward value according to the coverage rate of the operation code and the current number of steps, wherein the reward value is positively correlated with the coverage rate of the operation code and the current number of steps executed;

[0025] The reward value is used as a reward input from the simulated trading platform to the trading agent model.

[0026] Preferably, the transaction agent model is a deep Q learning network DQN model; the training process of the transaction agent model includes:

[0027] Obtain multiple smart contract codes and determine multiple function clusters based on each smart contract code;

[0028] Construct an action space A based on the multiple function clusters, construct a state space S based on the frequency of executed opcodes, the balance of the contract, and the frequency of each function cluster being selected, and construct a reward function R based on the opcode coverage and the current number of steps;

[0029] Initialize the network parameters of the DQN model ,in, , ;

[0030] Determine the selected function cluster through the DQN model and take action based on the selected function cluster , enter the state , and calculate the reward obtained by the action , update the network parameters Q:

[0031]

[0032] Among them, t is the current moment, t+1 is the next moment after the current moment, and refers to the learning rate, which is used to control the degree of difference between the previous Q value and the newly proposed Q value.

[0033] Preferably, the process of analyzing the log information to obtain the vulnerability detection result of the target smart contract includes:

[0034] Based on the preset vulnerability oracle, the log information is analyzed to obtain the vulnerability detection result of the target smart contract.

[0035] The second aspect of the present application provides a smart contract vulnerability detection device based on reinforcement learning, comprising:

[0036] An interface construction unit, used to construct an interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage rate;

[0037] a platform building unit, configured to connect the trained transaction agent model to the simulated transaction platform via the interactive interface, wherein the transaction agent model determines the function cluster selected in the transaction, and the simulated transaction platform responds to the selected function cluster and outputs corresponding log information;

[0038] A vulnerability detection unit, configured to analyze the log information and obtain vulnerability detection results for the target smart contract;

[0039] The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm.

[0040] A third aspect of the present application provides a smart contract vulnerability detection device based on reinforcement learning, comprising: a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement the various steps of the above-mentioned smart contract vulnerability detection method based on reinforcement learning.

[0043] In a fourth aspect, the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned reinforcement learning-based smart contract vulnerability detection method are implemented.

[0044] As can be seen from the above technical solution, this application first constructs an interactive interface between a transaction proxy model and a simulated trading platform based on the selected function clusters, the state of the smart contract, and the growth rate of opcode coverage. By clustering the functions in the smart contract into several function clusters, the action space for function cluster selection is simplified. By focusing on the growth rate of opcode coverage, functions in the smart contract's deep paths can be triggered. The trained transaction proxy model is then connected to the simulated trading platform through the interactive interface. The transaction proxy model determines the function clusters selected in the transaction to form the input for fuzz testing. The simulated trading platform responds to the selected function clusters and outputs corresponding log information to complete the fuzz test execution and output the results. Finally, by analyzing the log information, the vulnerability detection results of the target smart contract are obtained. The transaction proxy model is trained through unsupervised learning using a reinforcement learning algorithm, which can improve the efficiency of fuzz testing and explore deeper execution paths. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0046] Figure 1 A schematic diagram of a smart contract vulnerability detection method based on reinforcement learning disclosed in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the components of the reinforcement learning-based smart contract vulnerability detection method disclosed in an embodiment of the present application;

[0048] Figure 3 This is a system diagram of the reinforcement learning-based smart contract vulnerability detection method disclosed in an embodiment of the present application;

[0049] Figure 4 A schematic diagram illustrating the principle of reinforcement learning disclosed in the embodiments of this application is illustrated;

[0050] Figure 5 This is a schematic diagram of a smart contract vulnerability detection device based on reinforcement learning disclosed in an embodiment of the present application;

[0051] Figure 6 This is a schematic diagram of a smart contract vulnerability detection device based on reinforcement learning disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] The following describes the smart contract vulnerability detection method based on reinforcement learning provided by the embodiment of this application. Figures 1 to 3 The smart contract vulnerability detection method based on reinforcement learning provided in the embodiment of the present application may include the following steps:

[0054] Step S101: Construct an interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the operation code coverage.

[0055] This trading agent model is a neural network model based on reinforcement learning. Reinforcement learning is a field of machine learning that emphasizes how to act based on the environment to maximize expected returns. The focus is on finding a balance between exploration (of unknown areas) and utilization (of existing knowledge). This trading agent model can be a value-based algorithm model, such as Q-learning and DQN (Deep Q-learning Network), or a policy-based algorithm model, such as Trust Region Policy Optimization (TRPO) and Advantage Actor Critic (A2C).

[0056] The simulated trading platform is a virtual operating environment for smart contracts. A smart contract is a piece of code stored on the blockchain. Users can invoke a smart contract by initiating a specific transaction. They can also specify the function to be called and the function's parameters. Upon receiving the transaction, the smart contract will execute according to the code logic, and the results will be recorded on the blockchain.

[0057] This function cluster contains a class of functions with the same behavior or intention. For example, the function cluster for settings includes the settings of status and owner, etc.; the function cluster for transactions includes withdrawal operations and refund operations, etc.

[0058] Currently, there are many smart contract trading platforms. For example, the simulated trading platform can be developed based on Go Ethereum (geth). Furthermore, the RPC (Remote Procedure Call) method can be replaced by directly calling the geth API, thereby speeding up the testing process.

[0059] Step S102: Connect the trained transaction agent model to the simulated transaction platform through the interactive interface, the transaction agent model determines the function cluster selected in the transaction, and the simulated transaction platform responds to the selected function cluster and generates corresponding log information.

[0060] This step essentially models the smart contract fuzz testing process as Markov decision processes (MDPs) and transforms it into a reinforcement learning process. Reinforcement learning guides the fuzz testing tool (transaction agent model) in generating test inputs (function clusters and the transactions generated by them). This improves the transaction agent model's adaptability, enabling it to handle a wide range of smart contracts, rather than targeting specific types of smart contracts.

[0061] Among them, the core idea of ​​fuzz testing is to automatically or semi-automatically generate random data input into a program and monitor program anomalies to discover possible program errors. Fuzz testing is often used to detect security vulnerabilities in software or computer systems. For smart contracts, the input of the program is the transaction, so the fuzz testing of the smart contract in this application is actually to continuously let the smart contract execute transactions. Specifically, for the embodiment of this application, after the transaction agent model determines the function cluster selected in the transaction, it selects the corresponding function from the function cluster, further constructs a transaction, inputs it into the simulated trading platform, and then the simulated trading platform responds to the transaction and generates corresponding log information.

[0062] The transaction agent model is trained through unsupervised learning using a reinforcement learning algorithm. Through training, the transaction agent model can select a specific sequence of function clusters, enabling it to execute deep function calls on the smart contract to be tested. Figure 3 In the world of reinforcement learning, the agent interacts with the environment. The agent obtains the state from the environment and decides the action it wants to take. The environment will reward the agent according to its own logic. Rewards can be positive or negative. Specifically, corresponding to the application scenario of the embodiment of the present application, State corresponds to the properties of the smart contract, such as the frequency of executed opcodes, the balance of the contract, etc.; Action corresponds to the transaction, that is, the transaction generated by the function cluster selected in step S101; Reward corresponds to the growth rate of the opcode coverage rate, and the corresponding positive reward is given according to the value of the growth rate of the opcode coverage rate.

[0063] Step S103: Analyze the log information to obtain the vulnerability detection result of the target smart contract.

[0064] Among them, possible vulnerabilities in smart contracts may include dependence on block status, reentrancy vulnerabilities, dangerous delegate calls, dependence on external contracts, etc.

[0065] This application first constructs an interactive interface between a transaction proxy model and a simulated trading platform based on the selected function clusters, the state of the smart contract, and the growth rate of opcode coverage. By clustering the functions in the smart contract into several function clusters, the action space for function cluster selection is simplified. By focusing on the growth rate of opcode coverage, functions in the smart contract's deep paths can be triggered. The trained transaction proxy model is then connected to the simulated trading platform via the interactive interface. The transaction proxy model determines the function clusters selected in the transaction to form the input for fuzz testing. The simulated trading platform responds to the selected function clusters and outputs corresponding log information, completing the fuzz test execution and outputting the results. Finally, by analyzing the log information, vulnerability detection results for the target smart contract are obtained. The transaction proxy model is trained through unsupervised learning using a reinforcement learning algorithm, which can improve the efficiency of fuzz testing and discover deeper execution paths.

[0066] In some embodiments of the present application, the interaction interface mentioned in step S101 above may include a state interface, an action selection interface, and a reward feedback interface. The process of constructing the interaction interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage in step S101 above may include:

[0067] S1, based on the current state of the smart contract and multiple preset function clusters, builds the transaction agent model and the action selection interface of the simulation trading platform.

[0068] S2, based on the status of the smart contract, builds the status interface between the transaction agent model and the simulated trading platform.

[0069] S3, based on the growth rate of the opcode coverage, builds a reward feedback interface between the transaction agent model and the simulated trading platform.

[0070] By constructing the above-mentioned interactive interface, the fuzz testing of smart contracts is transformed into a reinforcement learning problem. The action, state, and reward functions are defined for the fuzz testing of smart contracts. The definition of these reinforcement learning elements plays a key role in the effectiveness of the fuzz testing, enabling the transaction agent model to find a better function call order, thereby excavating deeper execution paths and improving the efficiency of fuzz testing.

[0071] In some embodiments of the present application, the action selection interface in S1 is the action input from the transaction agent model to the simulated trading platform. The process of constructing the action selection interface between the transaction agent model and the simulated trading platform in S1 based on the current state of the smart contract and multiple preset function clusters may include:

[0072] S11, based on the current state of the smart contract, The greedy algorithm determines the target function cluster from the preset multiple function clusters, and determines the target function from the target function cluster. .

[0073] S12, randomly generated objective function The parameter x, the transaction initiator sender and the transaction amount amount are used to construct the action input t:

[0074]

[0075] Among them, the preset multiple function clusters are obtained by clustering the functions contained in different smart contracts through the K-means algorithm in advance, and each function cluster contains several functions with similar functions.

[0076] This application focuses solely on the selection of function clusters, specifically the order in which they are called. However, a smart contract transaction includes information such as the function and its parameters, the initiator, and the transaction amount. Therefore, once the transaction proxy model determines the selected function cluster, it can select a corresponding function from that cluster and randomly generate corresponding function parameters. This complete transaction is then input by the transaction proxy model into the simulated trading platform, which then responds and executes the transaction, returning the corresponding transaction results and generating log information.

[0077] In some embodiments of the present application, the state interface in S2 is the state input from the simulated trading platform to the transaction agent model. The process of constructing the state interface between the transaction agent model and the simulated trading platform in S2 based on the state of the smart contract may include:

[0078] S21, obtain the status of the smart contract, which may include the frequency of executed opcodes, the balance of the smart contract, and the frequency of each function cluster being selected.

[0079] Smart contracts are executed in the simulated trading platform in the form of bytecode. A string of bytecode is essentially a series of opcodes. Opcodes can be divided into opcodes for general stack operations (such as POP, PUSH, MLOAD, etc.), opcodes for general system operations (such as CREATE, CALL, RETURN, etc.), opcodes for arithmetic operations (such as MUL, SUB, DIV, etc.), and opcodes related to the environment (such as ADDRESS, BALANCE, CALLVALUE, etc.).

[0080] S22, converting the state into a feature vector, and using the feature vector as the state input from the simulated trading platform to the trading agent model.

[0081] The process of converting the state into a feature vector includes converting the state into an embedding method so that it can adapt to the input format of the neural network model.

[0082] In some embodiments of the present application, the reward feedback interface in S3 is the reward input from the simulated trading platform to the trading agent model. The process of constructing the reward feedback interface between the trading agent model and the simulated trading platform based on the growth rate of the opcode coverage rate in S3 may include:

[0083] S31, determining a reward value based on the coverage of the operation code and the current number of steps.

[0084] For example, the reward value can be determined by the following equation:

[0085]

[0086] in, is the regulating factor, is the current number of steps, is the coverage of the opcode.

[0087] S32: Using the reward value as a reward input from the simulated trading platform to the trading agent model.

[0088] The reward value is positively correlated with the opcode coverage and the number of steps currently executed. The opcode coverage can be the ratio of executed opcodes to all opcodes in the tested smart contract. The current number of steps can be the number of steps currently executed. The greater the number of steps tested, the greater the opcode coverage, indicating a deeper level of execution.

[0089] See also Figure 4 For the trading agent model mentioned in steps S101 and S102 above, the selected function cluster generates transactions (equivalent to the input in fuzz testing). The simulated trading platform processes these transactions and generates corresponding feedback (including status and rewards). To achieve the desired results, the trading agent model must first be trained, and this training process is actually an unsupervised learning process.

[0090] To avoid ambiguity, we define an interaction between fuzz testing and reinforcement learning here, including: state extraction, action selection, reward calculation, and network update.

[0091] Specifically, state extraction involves extracting the state of the smart contract, including the frequency of executed opcodes, the balance of the smart contract, and the frequency of each action being selected, and converting these states into feature vectors as the state input s of the reinforcement learning network.

[0092] Action selection is the selection of functions. According to the current state s, the reinforcement learning network uses a greedy algorithm with a probability of 1 - e (0 < e < 1) to select the function cluster (action) with the highest cumulative expectation. Among them, the 1 - e greedy algorithm selects the optimal action with a probability of 1 - e and randomly selects an action with a probability of e. After selecting the function cluster, a function is randomly selected from it, and together with randomly generated parameters, sender, and amount, a transaction is formed.

[0093] The process of executing a transaction is the process in which the smart contract is executed in the virtual trading platform and logs are output. Through the logs, the reward r and the new state s' can be calculated.

[0094] The process of updating the parameters of the network and the reinforcement learning network: The training process will retain the historical interaction sequence , randomly select from the state sequence S Perform backpropagation on the reinforcement learning network to update the network parameters.

[0095] Based on this, in some embodiments of the present application, the transaction agent model mentioned in the above steps S101 and S102 is a deep Q - learning network DQN model. The training process of this transaction agent model can include:

[0096] S1, Obtain multiple smart contract codes, and determine multiple function clusters based on each smart contract code.

[0097] Exemplarily, open - source codes of smart contracts can be collected from Etherscan as a data set, and then multiple function clusters are determined based on this data set. These multiple function clusters cover all functions in the data set.

[0098] By dividing each function into multiple function clusters and using the selection of function clusters as the basis for actions, the action space of reinforcement learning can be simplified. Exemplarily, based on the opcode frequency of functions, the functions included in the data set can be clustered by the K - means algorithm to obtain multiple function clusters.

[0099] S2, Construct an action space A according to the multiple function clusters, construct a state space S according to the executed opcode frequency, the balance of the contract, and the frequency of each function cluster being selected, and construct a reward function R according to the opcode coverage rate and the current number of steps.

[0100] S3, Initialize the network parameters of the DQN model , where, , .

[0101] S4, Determine the selected function cluster through the DQN model, and take actions according to the selected function cluster , enter the state , and calculate the reward obtained by the action , update the network parameters Q:

[0102]

[0103] Among them, t is the current moment, t+1 is the next moment after the current moment, and refers to the learning rate, which is used to control the degree of difference between the previous Q value and the newly proposed Q value.

[0104] By training the transaction agent model, it is possible to fully learn from previous test sequences (i.e., previously generated function call sequences) and adaptively generate function call sequences for deep paths, thereby increasing the probability of executing deep paths of smart contracts and detecting deep vulnerabilities in smart contracts, thereby solving the pain point that current tools have difficulty discovering deep vulnerabilities.

[0105] In some embodiments of the present application, the process of analyzing the log information in step S103 to obtain the vulnerability detection result of the target smart contract may include:

[0106] Based on the preset vulnerability oracle, the log information is analyzed to obtain the vulnerability detection results of the target smart contract.

[0107] The vulnerability oracle is defined based on the vulnerability pattern. The process of analyzing the log information based on the preset vulnerability oracle is the process of analyzing the log information based on the predefined vulnerability pattern.

[0108] For example, for vulnerabilities that depend on block status, the vulnerability pattern can be defined as follows: If there is an opcode for transfer (CALL) that depends on COINBASE, TIMESTAMP, NUMBER, DIFFICULTY, and / or GASLIMIT, then it is determined that the smart contract has a vulnerability that depends on block status for transfer.

[0109] For reentrancy vulnerabilities, which manifest as the ability of a smart contract to be repeatedly called to achieve the purpose of an attack, the vulnerability pattern can be defined as: based on the opcode and data flow, determine whether there is an operation that first transfers funds and then modifies the ledger. If so, it is determined that the target smart contract has a reentrancy vulnerability.

[0110] For dangerous delegate calls, the function call parameters are passed into the delegatecall parameters. The vulnerability pattern can be defined as follows: for the passed-in variable, if the variable itself or a related variable is used as a parameter of DelegateCall, then the smart contract is determined to have a dangerous delegate call vulnerability.

[0111] The following describes a smart contract vulnerability detection device based on reinforcement learning provided in an embodiment of the present application. The smart contract vulnerability detection device based on reinforcement learning described below and the smart contract vulnerability detection method based on reinforcement learning described above can refer to each other.

[0112] See Figure 5 The smart contract vulnerability detection device based on reinforcement learning provided in the embodiment of the present application may include:

[0113] An interface construction unit 21 is used to construct an interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the operation code coverage rate;

[0114] A platform building unit 22 is configured to connect the trained transaction agent model to the simulated transaction platform via the interactive interface, wherein the transaction agent model determines the function cluster selected in the transaction, and the simulated transaction platform responds to the selected function cluster and outputs corresponding log information;

[0115] A vulnerability detection unit 23 is used to analyze the log information and obtain vulnerability detection results of the target smart contract;

[0116] The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm.

[0117] In some embodiments of the present application, the interactive interface in the interface construction unit 21 includes a state interface, an action selection interface, and a reward feedback interface. The process of the interface construction unit 21 constructing the interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage rate may include:

[0118] Based on the current state of the smart contract and multiple preset function clusters, a transaction agent model and an action selection interface for the simulated trading platform are constructed;

[0119] According to the status of the smart contract, build the status interface between the transaction agent model and the simulated trading platform;

[0120] Based on the growth rate of the opcode coverage, a reward feedback interface between the transaction agent model and the simulated trading platform is constructed.

[0121] In some embodiments of the present application, the action selection interface in the interface construction unit 21 is the action input from the transaction agent model to the simulated trading platform; the process of the interface construction unit 21 constructing the action selection interface between the transaction agent model and the simulated trading platform based on the current state of the smart contract and multiple preset function clusters may include:

[0122] Based on the current state of the smart contract, The greedy algorithm determines the target function cluster from the preset multiple function clusters, and determines the target function from the target function cluster.

[0123] Randomly generate objective function The parameter x, the transaction initiator sender and the transaction amount, and construct the action input t

[0124]

[0125] Among them, the preset multiple function clusters are obtained by clustering the functions contained in different smart contracts through the K-means algorithm in advance, and each function cluster contains several functions with similar functions.

[0126] In some embodiments of the present application, the state interface in the interface construction unit 21 is the state input from the simulated trading platform to the transaction agent model; the process of the interface construction unit 21 constructing the state interface between the transaction agent model and the simulated trading platform based on the state of the smart contract may include:

[0127] Get the state of the smart contract, including the frequency of executed opcodes, the balance of the smart contract, and the frequency of each function cluster being selected;

[0128] The state is converted into a feature vector, and the feature vector is used as the state input from the simulated trading platform to the trading agent model.

[0129] In some embodiments of the present application, the reward feedback interface in the interface construction unit 21 is the reward input from the simulated trading platform to the trading agent model. The process of the interface construction unit 21 constructing the reward feedback interface between the trading agent model and the simulated trading platform based on the growth rate of the operation code coverage rate may include:

[0130] Determine a reward value according to the coverage rate of the operation code and the current number of steps, wherein the reward value is positively correlated with the coverage rate of the operation code and the current number of steps executed;

[0131] The reward value is used as a reward input from the simulated trading platform to the trading agent model.

[0132] In some embodiments of the present application, the transaction agent model mentioned in the interface construction unit 21 and the platform construction unit 22 is a deep Q learning network DQN model; the training process of the transaction agent model may include:

[0133] Obtain multiple smart contract codes and determine multiple function clusters based on each smart contract code;

[0134] The action space A is constructed based on the multiple function clusters. The state space S is constructed based on the frequency of the executed opcodes, the balance of the contract, and the frequency of each function cluster being selected. The reward function R is constructed based on the coverage of the opcodes and the current number of steps.

[0135] Initialize the network parameters of the DQN model ,in, ,

[0136] Determine the selected function cluster through the DQN model and take action based on the selected function cluster , enter the state , and calculate the reward obtained by the action , update the network parameters Q

[0137]

[0138] Among them, t is the current moment, t+1 is the next moment after the current moment, and refers to the learning rate, which is used to control the degree of difference between the previous Q value and the newly proposed Q value.

[0139] In some embodiments of the present application, the process of the vulnerability detection unit 23 analyzing the log information to obtain the vulnerability detection result of the target smart contract may include:

[0140] Based on the preset vulnerability oracle, the log information is analyzed to obtain the vulnerability detection result of the target smart contract.

[0141] The reinforcement learning-based smart contract vulnerability detection device provided in the embodiment of the present application can be applied to a reinforcement learning-based smart contract vulnerability detection device, such as a computer. Figure 6 The hardware structure diagram of the smart contract vulnerability detection device based on reinforcement learning is shown. Figure 6 The hardware structure of the smart contract vulnerability detection device based on reinforcement learning may include: at least one processor 31, at least one communication interface 32, at least one memory 33 and at least one communication bus 34.

[0142] In the embodiment of the present application, the number of the processor 31, the communication interface 32, the memory 33, and the communication bus 34 is at least one, and the processor 31, the communication interface 32, and the memory 33 communicate with each other through the communication bus 34;

[0143] The processor 31 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0144] The memory 32 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0145] The memory 33 stores a program, and the processor 31 can call the program stored in the memory 33, wherein the program is used to:

[0146] Based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage, an interactive interface between the transaction agent model and the simulated trading platform is constructed;

[0147] Connecting the trained transaction agent model to the simulated transaction platform through the interactive interface, the transaction agent model determining the function cluster selected in the transaction, and the simulated transaction platform responding to the selected function cluster and generating corresponding log information;

[0148] Analyze the log information to obtain vulnerability detection results of the target smart contract;

[0149] The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm.

[0150] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0151] An embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0152] Based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage, an interactive interface between the transaction agent model and the simulated trading platform is constructed;

[0153] Connecting the trained transaction agent model to the simulated transaction platform through the interactive interface, the transaction agent model determining the function cluster selected in the transaction, and the simulated transaction platform responding to the selected function cluster and generating corresponding log information;

[0154] Analyze the log information to obtain vulnerability detection results of the target smart contract;

[0155] The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm.

[0156] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0157] In summary:

[0158] This application first constructs an interactive interface between a transaction proxy model and a simulated trading platform based on the selected function clusters, the state of the smart contract, and the growth rate of opcode coverage. By clustering the functions in the smart contract into several function clusters, the action space for function cluster selection is simplified. By focusing on the growth rate of opcode coverage, functions in the smart contract's deep paths can be triggered. The trained transaction proxy model is then connected to the simulated trading platform via the interactive interface. The transaction proxy model determines the function clusters selected in the transaction to form the input for fuzz testing. The simulated trading platform responds to the selected function clusters and outputs corresponding log information, completing the fuzz test execution and outputting the results. Finally, by analyzing the log information, vulnerability detection results for the target smart contract are obtained. The transaction proxy model is trained through unsupervised learning using a reinforcement learning algorithm, which can improve the efficiency of fuzz testing and discover deeper execution paths.

[0159] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0160] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0161] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A smart contract vulnerability detection method based on reinforcement learning, characterized in that: include: Based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage, an interactive interface between the transaction agent model and the simulated trading platform is constructed; Connecting the trained transaction agent model to the simulated transaction platform through the interactive interface, the transaction agent model determining the function cluster selected in the transaction, and the simulated transaction platform responding to the selected function cluster and generating corresponding log information; The log information generation process is specifically as follows: After the transaction agent model determines the selected function cluster, it can select a corresponding function from the function cluster and randomly generate corresponding function parameters, which are used as a complete transaction and input into the simulated trading platform by the transaction agent model. The simulated trading platform responds to and executes the transaction, returns the corresponding transaction result, and generates log information. Analyze the log information to obtain vulnerability detection results of the target smart contract; The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm; The interactive interface includes a state interface, an action selection interface and a reward feedback interface; The process of constructing an interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the operation code coverage includes: Based on the current state of the smart contract and multiple preset function clusters, a transaction agent model and an action selection interface for the simulated trading platform are constructed; According to the status of the smart contract, build the status interface between the transaction agent model and the simulated trading platform; Based on the growth rate of opcode coverage, a reward feedback interface between the trading agent model and the simulated trading platform is constructed; The action selection interface is the action input from the transaction agent model to the simulated trading platform. The process of constructing the action selection interface between the transaction agent model and the simulated trading platform based on the current state of the smart contract and multiple preset function clusters includes: Based on the current state of the smart contract, The greedy algorithm determines the target function cluster from the preset multiple function clusters, and determines the target function from the target function cluster. ; Randomly generate objective function The parameter x, the transaction initiator sender and the transaction amount amount are used to construct the action input t: ； The preset multiple function clusters are obtained by clustering the functions contained in different smart contracts using the K-means algorithm in advance, and each function cluster contains several functions with similar functions; The state interface is the state input from the simulated trading platform to the transaction agent model. The process of constructing the state interface between the transaction agent model and the simulated trading platform based on the state of the smart contract includes: Get the state of the smart contract, including the frequency of executed opcodes, the balance of the smart contract, and the frequency of each function cluster being selected; Converting the state into a feature vector, and using the feature vector as a state input from the simulated trading platform to the trading agent model; The reward feedback interface is the reward input from the simulated trading platform to the trading agent model. The process of constructing the reward feedback interface between the trading agent model and the simulated trading platform based on the growth rate of the opcode coverage rate includes: Determine a reward value according to the coverage rate of the operation code and the current number of steps, wherein the reward value is positively correlated with the coverage rate of the operation code and the current number of steps executed; Using the reward value as a reward input from the simulated trading platform to the trading agent model; The transaction agent model is a deep Q learning network DQN model; the training process of the transaction agent model includes: Obtain multiple smart contract codes and determine multiple function clusters based on each smart contract code; Construct an action space A based on the multiple function clusters, construct a state space S based on the frequency of executed opcodes, the balance of the contract, and the frequency of each function cluster being selected, and construct a reward function R based on the opcode coverage and the current number of steps; Initialize the network parameters of the DQN model ,in, , ; Determine the selected function cluster through the DQN model and take action based on the selected function cluster , enter the state , and calculate the reward obtained by the action , update the network parameters Q: ； Among them, t is the current moment, t+1 is the next moment after the current moment, and refers to the learning rate, which is used to control the degree of difference between the previous Q value and the newly proposed Q value.

2. The method according to claim 1, characterized in that The process of analyzing the log information to obtain the vulnerability detection result of the target smart contract includes: Based on the preset vulnerability oracle, the log information is analyzed to obtain the vulnerability detection result of the target smart contract.

3. A smart contract vulnerability detection device based on reinforcement learning, applied to the smart contract vulnerability detection method based on reinforcement learning according to claim 1, characterized in that: include: An interface construction unit, used to construct an interactive interface between the transaction agent model and the simulated trading platform based on the selected function cluster, the state of the smart contract, and the growth rate of the opcode coverage rate; a platform building unit, configured to connect the trained transaction agent model to the simulated transaction platform via the interactive interface, wherein the transaction agent model determines the function cluster selected in the transaction, and the simulated transaction platform responds to the selected function cluster and outputs corresponding log information; A vulnerability detection unit, configured to analyze the log information and obtain vulnerability detection results for the target smart contract; The transaction agent model is obtained by unsupervised learning training using a reinforcement learning algorithm.

4. A smart contract vulnerability detection device based on reinforcement learning, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the smart contract vulnerability detection method based on reinforcement learning as described in any one of claims 1 to 2.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the smart contract vulnerability detection method based on reinforcement learning are implemented.

Images