A defense-in-depth security system and method based on blockchain and reinforcement learning

Abstract

The present invention discloses a defense-in-depth security system for blockchain and reinforcement learning, which relates to the field of information security technology. The system includes a device risk assessment module, an industrial firewall module, an industrial control network module, an industrial situational awareness module, an industrial control trust measurement module, a trusted access proxy module, an access user security identity infrastructure module, and a blockchain traceability and evidence recording module. The present invention also discloses a defense-in-depth security method for blockchain and reinforcement learning, which includes S100: preparation; S200: access user access to devices in the industrial control network; S300: terminal device access to the industrial control network; S400: assessment of industrial control network security situation; S500: real-time trust measurement; S600: dynamic adjustment of trust decisions; S700: execution of trust decisions; and S800: continuous monitoring and defense. The present invention improves the in-depth security and intelligence of industrial control networks.

Description

Technical Field

[0001] The present invention relates to the field of information security technologies, and in particular to a defense-in-depth security system and method based on blockchain and reinforcement learning. Background Art

[0002] Industrial control systems (ICS), also known as industrial control systems (ICS), refer to the collection of equipment, systems, networks, and controllers used to operate, control, and assist automated industrial production processes. They play a vital role in national infrastructure and are a critical resource for the nation's economy and people's livelihood. Attacks against ICS pose a significant threat to the normal operation of society, making the security of ICS systems extremely important. However, ICS systems encompass a broad range of logic, numerous application software, complex hardware, and diverse production equipment and environments. Once any component is attacked or compromised, it can impact the entire ICS system, causing irreparable damage. Furthermore, the advancement of computer and network technologies, the deep integration of informatization and industrialization, and the rapid growth of the Internet of Things have profoundly transformed traditional industrial models and industrial layouts, while also increasingly challenging the security landscape of ICS. ICS are evolving from a closed to an open model, and their widespread interconnectivity provides attackers with more attack paths, posing greater risks and threats to ICS security. Current ICS security approaches, characterized by perimeter protection, lack dynamic protection strategies, leaving ICS vulnerabilities exposed to attackers for a long time.

[0003] Faced with the complex and ever-changing challenges of industrial control security, the concept of defense-in-depth is gaining increasing attention. The fundamental principle of the defense-in-depth model is to organically combine information network security measures, deploying appropriate security measures tailored to the target, forming multiple lines of defense. These measures can mutually support and complement each other, minimizing the threat of attackers. However, existing defense-in-depth security models suffer from high costs and difficulties in achieving self-optimization of security policies and reliable record keeping, necessitating the introduction of new technical solutions.

[0004] Chinese patent authorization number CN112637220A, announcement date 2020.12.25, discloses a method and device for security protection of industrial control systems. The invention technology establishes an industrial control security baseline for the security protection issues of terminal equipment in the industrial control system, and combines the vulnerability database and threat intelligence database on the baseline to identify, predict, initialize and transfer security risks for the entire industrial control system, effectively improving the security level of the industrial control system. The invention constructs a security baseline at the terminal layer and network layer within the industrial control system, effectively protecting the security of terminals and networks connected to the industrial control system. However, the invention only constructs a layer of protection within the industrial control system, and the depth of the security system is insufficient. It cannot achieve effective protection in the face of more complex and changeable industrial control systems. In addition, the vulnerability database and threat intelligence database that have been established may also be attacked, resulting in data leakage, and there are no corresponding protection measures.

[0005] Therefore, technicians in this field are committed to developing a deep defense security system and method based on blockchain and reinforcement learning. Summary of the Invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the existing security defense architecture is insufficient in depth, has high security costs, and is difficult to achieve self-optimization and reliable recording of security policies.

[0007] The inventors have discovered through research that decentralized blockchain technology and reinforcement learning methods based on big data can effectively record data used in industrial control security defense processes and enable iterative optimization of security policies, making defense-in-depth measures more diverse and effective, thereby achieving fine-grained access control. The inventors have integrated blockchain technology and reinforcement learning methods into industrial control networks (including control networks and communication networks), and combined them with risk assessment and situational awareness methods to establish a comprehensive defense-in-depth security system and method. This improves the in-depth security of the security defense system, achieves both a secure and lightweight overall defense architecture, and self-optimizes the system's security policies and fault diagnosis. First, when the terminal device starts to enter the industrial control network, the terminal device information is fully scanned, and an authentication certificate for the terminal device is generated based on various risk factors. According to the level of the authentication certificate, the terminal device interacts and operates to enter different communication networks; secondly, the terminal device operation in the high-trust zero-trust domain adopts lightweight encryption and decryption and verification operations, and the terminal device operation in the low-trust zero-trust domain adopts more complex encryption and decryption and verification processing to achieve security and lightweight in parallel; then, various security parameters in the industrial control network are collected, and after pre-processing the data, a security perception model is constructed to conduct situation assessment and output security results; at the same time, the trust of the accessing user is measured in real time, and whether the accessing user is trustworthy is evaluated in real time. The incentive model is set in combination with the model data for training, and equipment failures in the industrial control network are intelligently diagnosed. Finally, based on the training results, optimized permission management rules are generated to optimize system security efficiency; in addition, key processes and data are recorded to achieve trusted evidence and auditing of security processes.

[0008] In one embodiment of the present invention, a defense-in-depth security system based on blockchain and reinforcement learning is provided, comprising:

[0009] The device risk assessment module performs risk assessment and certification on terminal devices and generates certification certificates of different levels;

[0010] The industrial firewall module isolates the industrial control network horizontally, has built-in industrial communication protocol parsing and filtering, and uses deep packet inspection technology and application layer communication tracking technology to intercept illegal instructions;

[0011] The industrial control network module, based on the zero-trust concept of "never trust, continuously verify," monitors user behavior in real time, measures trust, and dynamically adjusts access decisions.

[0012] The industrial situation awareness module extracts features from industrial control system data, builds an industrial control system situation model through machine learning methods, and outputs the current industrial control system security situation as the data basis for the incentive mechanism of the industrial control trust measurement module;

[0013] The user security identity infrastructure module manages user identities and permissions. Identity management manages the identity and lifecycle of users, while permissions management tracks and analyzes users. Identification refers to generating corresponding identity information for users to facilitate management.

[0014] The industrial control trust measurement module uses reinforcement learning methods to perform real-time trust measurement on the interaction between access users and devices in the industrial control network, and dynamically adjusts trust decisions based on the trust measurement results;

[0015] Trusted access agent module, as the execution module of trust decision, enables, monitors and terminates the connection between access users and devices in the industrial control network;

[0016] The blockchain traceability and evidence recording module receives and stores key data information that needs to be stored on the blockchain, publishes it to the blockchain network, and provides a security audit interface for regulators;

[0017] The equipment risk assessment module, industrial firewall module, industrial control network module, industrial situation awareness module, industrial control trust measurement module, and trusted access proxy module are sequentially connected in communication; the user security identity infrastructure module is connected in communication with the industrial situation awareness module and the industrial control trust measurement module; the equipment risk assessment module, industrial firewall module, industrial control trust measurement module, trusted access proxy module, and access user security identity infrastructure module are connected in communication with the blockchain traceability and evidence recording module respectively;

[0018] In response to a terminal device entering the industrial control network, the device risk assessment module conducts a comprehensive scan of the terminal device's information, performs risk assessment and authentication based on various risk factors, and generates a device authentication certificate. Based on the level of the authentication certificate, the terminal device accesses the corresponding zero-trust domain through the industrial firewall module to interact and operate, ensuring the security of the terminal device connected to the industrial control network.

[0019] In response to access requests from users, the industrial situation awareness module collects various security parameters of the industrial control network module, builds a security awareness model to perform situation assessment, and outputs security results to the industrial control trust measurement module. The industrial control trust measurement module measures the trustworthiness of the accessing user in real time and uses reinforcement learning methods to evaluate the accessing user's current trustworthiness in real time to ensure the accessing user's security.

[0020] The industrial control trust measurement module combines the model data of the industrial situation awareness module, sets the incentive model for training, intelligently diagnoses equipment failures in the industrial control network, and generates optimized permission management rules based on the training results; the blockchain traceability and evidence recording module records key data on the chain for trusted evidence and auditing of security processes.

[0021] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in the above embodiment, the terminal device includes all production, monitoring, and management equipment to be connected.

[0022] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in any of the above embodiments, the industrial control system data includes the operating status of equipment in the industrial control network, access user operation information, enterprise asset configuration information, and network topology structure.

[0023] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in any of the above embodiments, the industrial control system situation model data is stored on the chain for operation and management personnel to monitor the current security situation of the industrial control system.

[0024] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in any of the above embodiments, the device risk assessment module includes:

[0025] Device information database, which stores terminal device information, including terminal device type, model, software version, function, brand data, as well as terminal device vulnerability and patch security data;

[0026] The device scanning submodule is directly connected to the connected terminal device and scans the firmware information and communication messages of the terminal device to match the device information in the device information library;

[0027] The risk assessment submodule calculates the risk level of the terminal device based on the vulnerability and patch security data in the device information database and the preset risk factors;

[0028] The device registration submodule selects a suitable encryption algorithm for the connected terminal device based on the terminal device information transmitted by the risk assessment submodule, and encrypts and generates a key;

[0029] The certificate generation submodule generates authentication certificates of different levels based on the scanning information, initial risk level, and key generated by the device scanning submodule, risk assessment submodule, and device registration submodule, and sends the authentication certificates to the blockchain evidence recording module.

[0030] Optionally, in the in-depth defense security system based on blockchain and reinforcement learning in any of the above embodiments, the industrial control network module includes a high-trust zero-trust domain and a low-trust zero-trust domain. Terminal devices with high authentication certificate levels are low-risk devices and access the high-trust zero-trust domain network. Terminal devices with low authentication certificate levels are high-risk devices and access the low-trust zero-trust domain network.

[0031] Furthermore, in the defense-in-depth security system based on blockchain and reinforcement learning in the above embodiment, the low-trust zero-trust domain uses complex encryption and verification algorithms for the interactions between devices in the industrial control network and between devices in the industrial control network and access users to ensure the security of the industrial control system. The high-trust zero-trust domain uses lightweight encryption and verification algorithms for the interactions between devices in the industrial control network and between devices in the industrial control network and access users to ensure both the security of the industrial control system and efficiency.

[0032] Furthermore, in the defense-in-depth security system based on blockchain and reinforcement learning in the above embodiment, when a low-risk device detects malicious behavior or is attacked, its authentication certificate level is downgraded and it is converted to a high-risk device and accesses a low-trust zero-trust domain. Conversely, if the high-risk device is verified to be continuously trustworthy, its authentication certificate level is increased and converted to a low-risk device.

[0033] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in any of the above embodiments, the industrial control trust measurement module includes:

[0034] The reinforcement learning submodule uses the reinforcement learning method to measure the trust of access users in real time and transmits the measurement results to the dynamic access control submodule;

[0035] The dynamic access control submodule manages the permissions of access users. The measurement results output by the reinforcement learning submodule serve as the basis for permission management, granting different permissions to access users, and finally sending the permissions to the trusted access agent module for execution.

[0036] Furthermore, in the in-depth defense security system based on blockchain and reinforcement learning in the above embodiment, the reinforcement learning submodule extracts feature data as the state set S=(s1, s2, ...s n ), s1, s2, ... s n is the state at different moments, n∈N, N is a positive integer, the action set a=(a1,a2,a3,a4), a1,a2,a3,a4 correspond to four decisions: highly credible, credible, untrustworthy, and highly untrustworthy.

[0037] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in any of the above embodiments, the industrial situation awareness module includes:

[0038] The data preprocessing submodule preprocesses the industrial control system data, extracts the features required for the industrial control system situation model, and sends them to the model construction submodule. The industrial control system data includes the equipment operation status, system vulnerabilities, access user operation information, enterprise asset configuration information, and network topology structure in the industrial control network;

[0039] The model building sub-module constructs an industrial control system situation model through machine learning methods, obtains the weights of each parameter of the industrial control system situation model, and outputs security visualization data, models and situation judgment results for understanding the system operation safety status. The judgment results are stored on the chain and serve as the data basis for the reinforcement learning reward mechanism.

[0040] Optionally, in the defense-in-depth security system based on blockchain and reinforcement learning in any of the above embodiments, the trusted access agent module performs fine-grained access control based on the trust decision of the industrial control trust measurement module, while encrypting and protecting the industrial control network transmission traffic, and generating access record logs that are sent to the blockchain traceability and evidence recording module.

[0041] Furthermore, in the defense-in-depth security system based on blockchain and reinforcement learning in the above embodiment, the trusted access proxy module hides all business data behind the trusted access proxy module through single packet authorization (SPA) technology. Only users who meet the trust level requirements can access it, effectively protecting against DoS and DDoS attacks.

[0042] Based on any of the above embodiments, another embodiment of the present invention provides a defense-in-depth security method based on blockchain and reinforcement learning, including the following steps, wherein S200 and S300 are performed in any order:

[0043] S100: Preparation, including industrial control system situation model training and reinforcement learning DQN (Deep Q Learning) model training;

[0044] S200: An access user accesses a device in an industrial control network. The access user initiates an access request to the device in the industrial control network through a user security identity infrastructure module.

[0045] S300: The terminal device is connected to the industrial control network. The device risk assessment module performs risk assessment and authentication on the terminal device, generates a device authentication certificate, and connects the terminal device to the industrial control network module based on the level of the authentication certificate.

[0046] S400: Evaluate the industrial control network security situation. The industrial situation awareness module extracts the characteristic information of the equipment operating status in the industrial control network, judges the security status of the industrial control network in real time, and outputs it to the industrial control trust measurement module and the blockchain traceability and evidence storage module;

[0047] S500, real-time trust measurement. The industrial control trust measurement module uses the trained DQN model to measure the access user's behavior in real time. It uses the access user's status as input to determine whether the access user is allowed. The measurement result is sent to the dynamic access control submodule.

[0048] S600: Dynamically adjust the trust decision. The dynamic access control submodule maps the measurement results output by the industrial control trusted measurement module into access rights, makes a trust decision, and sends it to the trusted access proxy module for execution.

[0049] S700, executing the trust decision. In response to the trust decision sent by the dynamic access control submodule, the trusted access proxy module dynamically enables or terminates the connection between the access user and the device in the industrial control network in real time;

[0050] S800: Continuous monitoring and defense, repeating steps S400-S800.

[0051] Optionally, in the defense-in-depth security method based on blockchain and reinforcement learning in the above embodiment, step S100 includes:

[0052] S110, industrial control system situation model training;

[0053] S120, reinforcement learning DQN model training;

[0054] Optionally, in the defense-in-depth security method based on blockchain and reinforcement learning in the above embodiment, step S110 includes:

[0055] S111, data preprocessing: the data preprocessing submodule of the industrial situation awareness module preprocesses the collected equipment operation status, user operation information, enterprise asset configuration information, and network topology data in the industrial control network, extracts the features required for model construction, and sends them to the model construction submodule;

[0056] S112. Model construction. The model construction submodule of the industrial situation awareness module integrates and trains the input data features through machine learning, fits the weights of various parameters, outputs visual data, and obtains the industrial control system situation model.

[0057] Optionally, in the defense-in-depth security method based on blockchain and reinforcement learning in the above embodiment, step S120 includes:

[0058] S121, training Q network parameters, network input information set {S, a, r, t}, DQN reinforcement learning training, the current time is t, the action reward during training is r, is the change value of the situation awareness module output, the value function is Q; the initial selection uses random probability to select the action and is recorded as a t , subsequent actions are selected according to the principle of maximizing the value function, and the calculation formula is as follows:

[0059] a t =argmaxQ(S t ,a) (1)

[0060] Among them St is the state at time t, a t The action selected at time t. After calculating the action probability and making the action selection through the Q network, the industrial situation awareness module updates the current reward value r and the next state S in real time. t+1 ;

[0061] S122, train the tarket-Q network parameters, and the actual Q value is calculated by the tarket-Q network simulation, the formula is as follows:

[0062]

[0063] Where λ∈[0,1] is the discount coefficient, which is used to balance the immediate value function and the future value function. is the weight parameter of the tarket-Q network. The tarket-Q network uses the Bellman equation to calculate the loss function L(ω), and uses the stochastic gradient descent method to update the Q network parameter ω for L(ω), and sets the two network parameters to be synchronized once every two iterations.

[0064]

[0065] Where ω is the Q network weight parameter;

[0066] S123, update the network information set and synchronize the network parameters, each time the reinforcement learning submodule interacts with the visiting user, the sample (S t ,a t ,r t ,S t+1 ) will be stored in the network memory pool as the sample for the next round of training of the tarket-Q network, disrupting the correlation of the samples, and tarket-Q network synchronizes Q network parameters every certain number of iterations;

[0067] S124. Randomly select samples from the memory pool. After the current training round is completed, randomly select samples from the memory pool as supplements to continue training. Repeat steps S121-S214 until the loss function L(ω) is minimized.

[0068] Optionally, in the defense-in-depth security method based on blockchain and reinforcement learning in any of the above embodiments, step S300 includes:

[0069] S310, device scanning, the device scanning submodule scans the firmware information and communication messages of the terminal device;

[0070] S320, device information matching, matching device information in the device information database according to the firmware information and the communication message;

[0071] S330, risk level calculation, the risk assessment submodule calculates the risk level of the terminal device based on the vulnerability and patch security data in the device information database combined with the preset risk factors;

[0072] S340, key generation: the device registration submodule selects a suitable encryption algorithm for the connected terminal device and encrypts and generates a key;

[0073] S350, authentication certificate generation. The certificate generation submodule generates authentication certificates of different levels based on the scanning information, initial risk level, and key of the connected terminal device, and sends the certificate information to the blockchain evidence recording module.

[0074] The present invention integrates blockchain technology and reinforcement learning methods into industrial control networks, and combines risk assessment and situational awareness methods to establish a complete in-depth defense security system and method, thereby improving the in-depth security of the security defense system, achieving the security and lightweight parallelization of the overall defense architecture, and self-optimization of the system's security strategy and fault diagnosis; based on the reinforcement learning method, it realizes the detection and judgment of the entire security defense process, and continuously optimizes and improves the security strength of the system; based on blockchain technology, it realizes the trusted storage and audit of key data of the entire process security record. The present invention improves the in-depth security and intelligence of the industrial control network.

[0075] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is a structural diagram illustrating a defense-in-depth security system based on blockchain and reinforcement learning according to an exemplary embodiment;

[0077] Figure 2 is a flow chart illustrating a defense-in-depth security method based on blockchain and reinforcement learning according to an exemplary embodiment;

[0078] Figure 3 is a flow chart illustrating a defense-in-depth secure reinforcement learning DQN model training based on blockchain and reinforcement learning according to an exemplary embodiment;

[0079] Figure 4 is a flow chart illustrating risk assessment of a defense-in-depth security approach based on blockchain and reinforcement learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0080] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0081] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. To enhance clarity, the thickness of components in some places in the drawings is schematically exaggerated.

[0082] The inventors designed a defense-in-depth security system based on blockchain and reinforcement learning, such as Figure 1 As shown, including:

[0083] The equipment risk assessment module performs risk assessment and certification on terminal equipment and generates certification certificates of different levels. Terminal equipment includes all production, monitoring, and management equipment to be connected. The equipment risk assessment module includes:

[0084] Device information database, which stores terminal device information, including terminal device type, model, software version, function, brand data, as well as terminal device vulnerability and patch security data;

[0085] The device scanning submodule is directly connected to the connected terminal device and scans the firmware information and communication messages of the terminal device to match the device information in the device information library;

[0086] The risk assessment submodule calculates the risk level of the terminal device based on the vulnerability and patch security data in the device information database and the preset risk factors;

[0087] The device registration submodule selects a suitable encryption algorithm for the connected terminal device based on the terminal device information transmitted by the risk assessment submodule, and encrypts and generates a key;

[0088] The certificate generation submodule generates authentication certificates of different levels based on the scanning information, initial risk level, and key generated by the device scanning submodule, risk assessment submodule, and device registration submodule, and sends the authentication certificates to the blockchain evidence recording module.

[0089] The industrial firewall module isolates the industrial control network horizontally, has built-in industrial communication protocol parsing and filtering, and uses deep packet inspection technology and application layer communication tracking technology to intercept illegal instructions;

[0090] The industrial control network module, based on the zero-trust concept of "never trust, continuous verification", monitors the behavior of accessing users in real time and measures trust, and dynamically adjusts access decisions. The industrial control network module includes high-trust zero-trust domains and low-trust zero-trust domains. Terminal devices with high authentication certificate levels are low-risk devices and access the high-trust zero-trust domain network, while terminal devices with low authentication certificate levels are high-risk devices and access the low-trust zero-trust domain network. The low-trust zero-trust domain uses complex encryption and verification algorithms for interactions between devices in the industrial control network and between devices in the industrial control network and accessing users to ensure the security of the industrial control system. The high-trust zero-trust domain uses lightweight encryption and verification algorithms for interactions between devices in the industrial control network and between devices in the industrial control network and accessing users, taking into account the security of the industrial control system while ensuring efficiency. When a low-risk device detects malicious behavior or is attacked, its authentication certificate level is downgraded and it is converted to a high-risk device and accesses the low-trust zero-trust domain. Conversely, if the high-risk device is verified to be continuously trustworthy, its authentication certificate level is increased and it is converted to a low-risk device.

[0091] The industrial situation awareness module extracts features from industrial control system data, constructs an industrial control system situation model through machine learning methods, and outputs the current industrial control system security situation as the data basis for the incentive mechanism of the industrial control trust measurement module. The industrial control system data includes the operating status of equipment in the industrial control network, access user operation information, enterprise asset configuration information, and network topology structure. The model data is stored on the chain for operators and managers to monitor the current security situation of the industrial control system. The industrial situation awareness module includes:

[0092] The data preprocessing submodule preprocesses the industrial control system data, extracts the features required for the industrial control system situation model, and sends them to the model construction submodule. The industrial control system data includes the equipment operation status, system vulnerabilities, access user operation information, enterprise asset configuration information, and network topology structure in the industrial control network;

[0093] The model building sub-module constructs an industrial control system situation model through machine learning methods, obtains the weights of each parameter of the industrial control system situation model, and outputs security visualization data, models and situation judgment results for understanding the system operation safety status. The judgment results are stored on the chain and serve as the data basis for the reinforcement learning reward mechanism.

[0094] The user security identity infrastructure module manages user identities and permissions. Identity management manages the identity and lifecycle of users, while permissions management tracks and analyzes users. Identification refers to generating corresponding identity information for users to facilitate management.

[0095] The Industrial Control Trust Measurement Module uses reinforcement learning to perform real-time trust measurement on the interactions between access users and devices in the industrial control network, and dynamically adjusts trust decisions based on the trust measurement results. The Industrial Control Trust Measurement Module includes:

[0096] The reinforcement learning submodule uses the reinforcement learning method to measure the trust of access users in real time and transmits the measurement results to the dynamic access control submodule; the reinforcement learning submodule extracts feature data through the convolutional network as the state set S = (s1, s2, ... s n ), s1, s2, ... s n is the state at different moments, n∈N, N is a positive integer, the action set a=(a1,a2,a3,a4), a1,a2,a3,a4 correspond to four decisions: highly credible, credible, uncredible, highly uncredible;

[0097] The dynamic access control submodule manages the permissions of access users. The measurement results output by the reinforcement learning submodule serve as the basis for permission management, granting different permissions to access users, and finally sending the permissions to the trusted access agent module for execution.

[0098] The trusted access proxy module, as the execution module of trust decisions, enables, monitors, and terminates the connection between access users and devices in the industrial control network. The trusted access proxy module performs fine-grained access control based on the trust decisions of the industrial control trust measurement module, while encrypting and protecting the industrial control network transmission traffic, and generating access records and sending logs to the blockchain traceability and evidence recording module. The trusted access proxy module uses single packet authorization (SPA) technology to hide all business data behind the trusted access proxy module. Only access users who meet the trust level requirements can access it, effectively protecting against DoS and DDoS attacks.

[0099] The blockchain traceability and evidence recording module receives and stores key data information that needs to be stored on the blockchain, publishes it to the blockchain network, and provides a security audit interface for regulators;

[0100] The equipment risk assessment module, industrial firewall module, industrial control network module, industrial situation awareness module, industrial control trust measurement module, and trusted access proxy module are sequentially connected in communication; the user security identity infrastructure module is connected in communication with the industrial situation awareness module and the industrial control trust measurement module; the equipment risk assessment module, industrial firewall module, industrial control trust measurement module, trusted access proxy module, and access user security identity infrastructure module are connected in communication with the blockchain traceability and evidence recording module respectively;

[0101] In response to a terminal device entering the industrial control network, the device risk assessment module conducts a comprehensive scan of the terminal device's information, performs risk assessment and authentication based on various risk factors, and generates a device authentication certificate. Based on the level of the authentication certificate, the terminal device accesses the corresponding zero-trust domain through the industrial firewall module to interact and operate, ensuring the security of the terminal device connected to the industrial control network.

[0102] In response to access requests from users, the industrial situation awareness module collects various security parameters of the industrial control network module, builds a security awareness model to perform situation assessment, and outputs security results to the industrial control trust measurement module. The industrial control trust measurement module measures the trustworthiness of the accessing user in real time and uses reinforcement learning methods to evaluate the accessing user's current trustworthiness in real time to ensure the accessing user's security.

[0103] The industrial control trust measurement module combines the model data of the industrial situation awareness module, sets the incentive model for training, intelligently diagnoses equipment failures in the industrial control network, and generates optimized permission management rules based on the training results; the blockchain traceability and evidence recording module records key data on the chain for trusted evidence and auditing of security processes.

[0104] Based on the above embodiments, the inventor provides a defense-in-depth security method based on blockchain and reinforcement learning, such as Figure 2 As shown, the following steps are included, wherein S200 and S300 are performed in any order:

[0105] S100: Preparation, including industrial control system situation model training and reinforcement learning DQN (Deep Q Learning) model training; specifically, including:

[0106] S110, Industrial Control System Situation Model Training; including:

[0107] S111, data preprocessing: the data preprocessing submodule of the industrial situation awareness module preprocesses the collected equipment operation status, user operation information, enterprise asset configuration information, and network topology data in the industrial control network, extracts the features required for model construction, and sends them to the model construction submodule;

[0108] S112. Model construction. The model construction submodule of the industrial situation awareness module integrates and trains the input data features through machine learning, fits the weights of various parameters, outputs visual data, and obtains the industrial control system situation model.

[0109] S120, reinforcement learning DQN model training, such as Figure 3 As shown, including:

[0110] S121, training Q network parameters, network input information set {S, a, r, t}, DQN reinforcement learning training, the current time is t, the action reward during training is r, is the change value of the situation awareness module output, the value function is Q; the initial selection uses random probability to select the action and is recorded as a t , subsequent actions are selected according to the principle of maximizing the value function, and the calculation formula is as follows:

[0111] a t =argmaxQ(S t ,a) (1)

[0112] Among them S t is the state at time t, a t The action selected at time t. After calculating the action probability and making the action selection through the Q network, the industrial situation awareness module updates the current reward value r and the next state S in real time. t+1 ;

[0113] S122, train the tarket-Q network parameters, and the actual Q value is calculated by the tarket-Q network simulation, the formula is as follows:

[0114]

[0115] Where λ∈[0,1] is the discount coefficient, which is used to balance the immediate value function and the future value function. is the weight parameter of the tarket-Q network. The tarket-Q network uses the Bellman equation to calculate the loss function L(ω), and uses the stochastic gradient descent method to update the Q network parameter ω for L(ω), and sets the two network parameters to be synchronized once every two iterations.

[0116]

[0117] Where ω is the Q network weight parameter;

[0118] S123, update the network information set and synchronize the network parameters, each time the reinforcement learning submodule interacts with the visiting user, the sample (S t ,a t ,r t ,S t+1 ) will be stored in the network memory pool as the sample for the next round of training of the tarket-Q network, disrupting the correlation of the samples, and tarket-Q network synchronizes Q network parameters every certain number of iterations;

[0119] S124. Randomly select samples from the memory pool. After the current training round is completed, randomly select samples from the memory pool as supplements to continue training. Repeat steps S121-S214 until the loss function L(ω) is minimized.

[0120] S200: An access user accesses a device in an industrial control network. The access user initiates an access request to the device in the industrial control network through a user security identity infrastructure module.

[0121] S300, the terminal device is connected to the industrial control network, the device risk assessment module performs risk assessment and authentication on the terminal device, generates a device authentication certificate, and connects to the industrial control network module according to the level of the authentication certificate; Figure 4 As shown, including:

[0122] S310, device scanning, the device scanning submodule scans the firmware information and communication messages of the terminal device;

[0123] S320, device information matching, matching device information in the device information database according to the firmware information and the communication message;

[0124] S330, risk level calculation, the risk assessment submodule calculates the risk level of the terminal device based on the vulnerability and patch security data in the device information database combined with the preset risk factors;

[0125] S340, key generation: the device registration submodule selects a suitable encryption algorithm for the connected terminal device and encrypts and generates a key;

[0126] S350, authentication certificate generation. The certificate generation submodule generates authentication certificates of different levels based on the scanning information, initial risk level, and key of the connected terminal device, and sends the certificate information to the blockchain evidence recording module.

[0127] S400: Evaluate the industrial control network security situation. The industrial situation awareness module extracts the characteristic information of the equipment operating status in the industrial control network, judges the security status of the industrial control network in real time, and outputs it to the industrial control trust measurement module and the blockchain traceability and evidence storage module;

[0128] S500, real-time trust measurement. The industrial control trust measurement module uses the trained DQN model to measure the access user's behavior in real time. It uses the access user's status as input to determine whether the access user is allowed. The measurement result is sent to the dynamic access control submodule.

[0129] S600: Dynamically adjust the trust decision. The dynamic access control submodule maps the measurement results output by the industrial control trusted measurement module into access rights, makes a trust decision, and sends it to the trusted access proxy module for execution.

[0130] S700, executing the trust decision. In response to the trust decision sent by the dynamic access control submodule, the trusted access proxy module dynamically enables or terminates the connection between the access user and the device in the industrial control network in real time;

[0131] S800: Continuous monitoring and defense, repeating steps S400-S800.

[0132] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A defense-in-depth security system based on blockchain and reinforcement learning, characterized by: include: The device risk assessment module performs risk assessment and certification on terminal devices and generates certification certificates of different levels; The industrial firewall module isolates the industrial control network horizontally, has built-in industrial communication protocol parsing and filtering, and uses deep packet inspection technology and application layer communication tracking technology to intercept illegal instructions; The industrial control network module monitors user behavior and trust metrics in real time, and dynamically adjusts access decisions. The industrial control network module includes a high-trust zero-trust domain and a low-trust zero-trust domain. Terminal devices with high authentication certificate levels are low-risk devices and access the high-trust zero-trust domain network. Terminal devices with low authentication certificate levels are high-risk devices and access the low-trust zero-trust domain network. The low-trust zero-trust domain uses complex encryption and verification algorithms for the interactions between devices in the industrial control network and between devices in the industrial control network and the access users to ensure the security of the industrial control system. The high-trust zero-trust domain uses lightweight encryption and verification algorithms for the interactions between devices in the industrial control network and between devices in the industrial control network and the access users, taking into account the security of the industrial control system while ensuring efficiency. When the low-risk device detects malicious behavior or is attacked, its authentication certificate level is reduced and it is converted to a high-risk device and connected to the low-trust zero-trust domain. Conversely, the high-risk device is continuously authenticated and its authentication certificate level is increased to a low-risk device. The industrial situation awareness module extracts features from industrial control system data, builds an industrial control system situation model through machine learning methods, and outputs the current industrial control system security situation as the data basis for the incentive mechanism of the industrial control trust measurement module; The user security identity infrastructure module performs identity management and rights management of the accessing user, manages the identity and identity lifecycle of the accessing user through identity management, and tracks and analyzes the accessing user through rights management; An industrial control trust measurement module performs real-time trust measurement on the interaction between the access user and the equipment in the industrial control network through a reinforcement learning method, and dynamically adjusts the trust decision based on the trust measurement results; The industrial control trust measurement module includes: The reinforcement learning submodule uses the reinforcement learning method to perform real-time trust measurement on the access user and transmits the measurement results to the dynamic access control submodule; the reinforcement learning submodule extracts feature data as a state set through a convolutional network , For different time states, , N is a positive integer, the action set , They correspond to four types of decisions: highly credible, credible, uncredible, and highly uncredible; The dynamic access control submodule manages the permissions of access users. The measurement results output by the reinforcement learning submodule serve as the basis for permission management, granting different permissions to the access users, and ultimately sending the permissions to the trusted access proxy module for execution; A trusted access agent module, serving as an execution module for trust decisions, enables, monitors, and terminates the connection between the access user and the device in the industrial control network; The blockchain traceability and evidence recording module receives and stores key data information that needs to be stored on the blockchain, publishes it to the blockchain network, and provides a security audit interface for regulators; The device risk assessment module, the industrial firewall module, the industrial control network module, the industrial situation awareness module, the industrial control trust measurement module, and the trusted access proxy module are sequentially communicated and connected, the user security identity infrastructure module is communicated and connected with the industrial situation awareness module and the industrial control trust measurement module, and the device risk assessment module, the industrial firewall module, the industrial control trust measurement module, the trusted access proxy module, and the access user security identity infrastructure module are respectively communicated and connected with the blockchain traceability and evidence recording module; In response to the terminal device entering the industrial control network, the device risk assessment module comprehensively scans the information of the terminal device, performs risk assessment and authentication based on various risk factors, and generates a device authentication certificate. The terminal device accesses the corresponding zero-trust domain through the industrial firewall module according to the level of the authentication certificate, interacts and operates, and ensures the security of the terminal device accessing the industrial control network; In response to the access request of the access user, the industrial situation awareness module collects various security parameters of the industrial control network module, constructs a security awareness model to perform situation assessment, outputs a security result and sends it to the industrial control trust measurement module. The industrial control trust measurement module measures the trustworthiness of the access user in real time and uses a reinforcement learning method to evaluate whether the access user is currently trustworthy in real time to ensure the security of the access user; The industrial control trust measurement module combines the model data of the industrial situation awareness module, sets an incentive model for training, intelligently diagnoses equipment failures in the industrial control network, and generates optimized permission management rules based on the training results; the blockchain traceability and evidence recording module records key data of on-chain evidence for trusted evidence and auditing of security processes.

2. The blockchain and reinforcement learning defense-in-depth security system according to claim 1, characterized in that: The equipment risk assessment module includes: Device information database, which stores terminal device information, including terminal device type, model, software version, function, brand data, as well as terminal device vulnerability and patch security data; A device scanning submodule is directly connected to the connected terminal device, and scans the firmware information and communication messages of the terminal device to match the device information in the device information library; A risk assessment submodule, which calculates the risk level of the terminal device based on the vulnerability and patch security data in the device information library combined with a preset risk factor; The device registration submodule selects a suitable encryption algorithm for the connected terminal device based on the terminal device information transmitted by the risk assessment submodule, and performs encryption to generate a key; The certificate generation submodule generates authentication certificates of different levels based on the scanning information, initial risk level, and key generated by the device scanning submodule, the risk assessment submodule, and the device registration submodule, and sends the authentication certificates to the blockchain evidence recording module.

3. The blockchain and reinforcement learning defense-in-depth security system according to claim 2, wherein: The industrial situation awareness module includes: A data preprocessing submodule preprocesses the industrial control system data, extracts the features required for the industrial control system situation model, and sends them to the model construction submodule. The industrial control system data includes the equipment operation status, system vulnerabilities, access user operation information, enterprise asset configuration information, and network topology structure in the industrial control network; The model building sub-module constructs an industrial control system situation model through machine learning methods, obtains the weights of each parameter of the industrial control system situation model, and outputs security visualization data, models and situation judgment results for understanding the system operation safety status. The judgment results are stored on the chain and serve as the data basis for the reinforcement learning reward mechanism.

4. A defense-in-depth security method based on blockchain and reinforcement learning, using the blockchain and reinforcement learning defense-in-depth security system as claimed in claim 3, characterized in that: The following steps are included, wherein S200 and S300 are performed in any order: S100: Preparation, including industrial control system situation model training and reinforcement learning DQN model training; specifically: S110, Industrial Control System Situation Model Training; specifically includes: S111, data preprocessing: the data preprocessing submodule of the industrial situation awareness module preprocesses the collected equipment operating status, user operation information, enterprise asset configuration information, and network topology data in the industrial control network, extracts features required for model construction, and sends the features to the model construction submodule; S112, model construction, the model construction submodule of the industrial situation awareness module integrates and trains the input data features through machine learning, fits the weights of various parameters, outputs visual data, and obtains the industrial control system situation model; S120, reinforcement learning DQN model training, specifically including: S121, training Q network parameters, network input information set , conduct reinforcement learning training of DQN, the current moment is , the action reward during training is , is the change value output by the situation awareness module, and the value function is ; When the action is initially selected, random probability is used to select the action and recorded as , subsequent actions are selected according to the principle of maximizing the value function, and the calculation formula is as follows: in for The state of the moment, for The action selected at each moment, after the Q network calculates the action probability and makes the action selection, the industrial situation awareness module updates the current reward value in real time And the next status ; S122, training tarket-Q network parameters, while the actual The value is calculated by tarket-Q network simulation, and the formula is as follows: in is the discount factor, which is used to balance the immediate value function and the future value function. is the weight parameter of the tarket-Q network, which uses the Bellman equation to calculate the loss function ,right Update the Q network parameters using stochastic gradient descent , and set the two network parameters to be synchronized once every 2 iterations, in is the Q network weight parameter; S123, update the network information set and synchronize the network parameters, each time the reinforcement learning submodule interacts with the visiting user to obtain the sample It will be stored in the network memory pool as the sample for the next round of training of the tarket-Q network, disrupting the correlation of the samples, and tarket-Q network will synchronize the Q network parameters every certain number of iterations; S124, randomly select samples from the memory pool. After the current training round is over, randomly select samples from the memory pool as supplements to continue training, and repeat steps S121-S214 until the loss function to the lowest possible level; S200: An access user accesses a device in an industrial control network, and the access user initiates an access request to the device in the industrial control network through the user security identity infrastructure module; S300: The terminal device is connected to the industrial control network. The device risk assessment module performs risk assessment and authentication on the terminal device, generates a device authentication certificate, and connects the terminal device to the industrial control network module according to the level of the authentication certificate. S400: Assessing the security status of the industrial control network. The industrial situation awareness module extracts characteristic information about the operating status of devices in the industrial control network, determines the security status of the industrial control network in real time, and outputs the information to the industrial control trust measurement module and the blockchain traceability and evidence storage module. S500, real-time trust measurement: the industrial control trust measurement module measures the behavior of the accessing user in real time through the trained DQN model, takes the accessing user's status as input, determines whether the accessing user is allowed to access, and sends the measurement result to the dynamic access control submodule; S600: Dynamically adjust the trust decision. The dynamic access control submodule maps the measurement results output by the industrial control trust measurement module into access rights, makes a trust decision, and sends it to the trusted access proxy module for execution. S700: Execute a trust decision. In response to the trust decision sent by the dynamic access control submodule, the trusted access proxy module dynamically enables or terminates the connection between the access user and the device in the industrial control network in real time. S800: Continuous monitoring and defense, repeating steps S400-S800.

5. The in-depth defense security method based on blockchain and reinforcement learning as claimed in claim 4, characterized in that: The step S300 includes: S310, device scanning, the device scanning submodule scans the firmware information and communication messages of the terminal device; S320: Match device information in the device information database according to the firmware information and the communication message. S330, risk level calculation, the risk assessment submodule calculates the risk level of the terminal device based on the vulnerability and patch security data in the device information database combined with the preset risk factors; S340, key generation: the device registration submodule selects a suitable encryption algorithm for the connected terminal device and encrypts and generates a key; S350, authentication certificate generation. The certificate generation submodule generates authentication certificates of different levels based on the scanning information, initial risk level, and key of the connected terminal device, and sends the certificate information to the blockchain evidence recording module.

Images