Network security protection method based on reinforcement learning and related equipment

Abstract

The invention discloses a network security protection method based on reinforcement learning and related equipment, which are suitable for network security protection and toughness improvement of a physical information system. In a preset heterogeneous executor pool arranged on the defense platform, generating a heterogeneous executor combination according to preset redundancy to obtain a heterogeneous executor combination set; updating the heterogeneous executor combination set according to the CVE vulnerability database to obtain a target heterogeneous executor combination set; when an abnormal threat event is identified, calling a heterogeneous executor combination in a target heterogeneous executor combination set to defend the abnormal threat event based on a heuristic rule, and obtaining a defense result; and generating a protection strategy for the abnormal threat event based on the defense result and an EXP3 algorithm, and executing a protection action according to the protection strategy. The attack effect can be perceived in real time in a continuously changing attack environment, the defense resource distribution can be autonomously adjusted, the optimal scheduling scheme can be dynamically evolved, and the resistance to known and unknown security threats can be improved.

Description

Technical Field

[0001] The present invention relates to the field of information security and artificial intelligence technology, and specifically to a network security protection method and related equipment based on reinforcement learning. Background Art

[0002] With the continuous development of information and network technologies, cyber-physical systems (CPS) have been widely applied in multiple critical infrastructure sectors, including power, transportation, energy, and manufacturing. CPS integrates computing, networking, and physical processes, and possesses a high degree of autonomy and real-time capabilities. Their security is crucial for ensuring public safety and economic operations.

[0003] However, CPS systems face increasingly complex and intense cybersecurity threats. In recent years, attacks against CPS systems have steadily increased, with evolving attack methods. Attacks targeting system vulnerabilities, configuration flaws, and supply chain vulnerabilities are particularly common. These attacks can not only cause system failures but also physical damage and significant social harm. For example, ransomware, worms, and targeted APT attacks have all severely impacted critical control systems.

[0004] Traditional network security protection methods mainly rely on passive defense measures such as static rules, regular vulnerability repairs, or single redundant resource configuration. These solutions generally have the following flaws: Insufficient adaptability: Static security policies are difficult to cope with dynamic changes in attack methods. Once an attacker breaks through the current defense, the system is often unable to adjust the policy in time, resulting in excessively long risk exposure time.

[0005] Weak protection capabilities against unknown threats: Traditional defenses focus more on known attack signatures and have limited capabilities in identifying and responding to unknown, variant, and combined attacks.

[0006] Low resource utilization efficiency: The redundant resource allocation scheme lacks the ability to schedule attacks in response to the situation, which can easily lead to resource waste or insufficient defense capabilities at critical moments.

[0007] Lack of flexible recovery mechanisms: When a breach occurs, existing systems often cannot achieve rapid switching and recovery, lack resilience, and significantly affect system availability.

[0008] To address these shortcomings, dynamic defense architectures, such as mobile target defense (MTD) and mimic defense, have gradually attracted research attention in recent years. Dynamic Heterogeneous Redundancy (DHR) significantly improves the system's immunity to both known and unknown vulnerabilities through heterogeneous resources and redundant configurations. However, challenges remain in resource scheduling and intelligent self-adaptation. For example, current DHR systems' post-attack switching mechanisms mostly rely on fixed, random, or simple heuristic algorithms, lacking intelligent strategy optimization for complex attack-defense games and the dynamic evolution of attack behavior.

[0009] In recent years, the application of artificial intelligence and machine learning technologies in network security has matured. Reinforcement learning, as an online adaptive algorithm, can dynamically optimize decision-making strategies based on environmental feedback in unknown or changing environments, demonstrating excellent adaptability and strategy evolution. However, existing reinforcement learning methods are often limited to single-layer, single-objective optimization, making them difficult to meet the practical requirements of multi-resource, multi-objective, and multi-level adaptive scheduling in CPS systems. Summary of the Invention

[0010] The purpose of the present invention is to provide a network security protection method and related equipment based on reinforcement learning, to realize the autonomous optimization of the defense strategy of perception-decision-feedback integration under the dynamic heterogeneous redundant architecture, to solve the technical problems such as poor adaptability and one-sided resilience measurement in the existing technology, and to enhance the self-repair and continuous service capabilities of the physical information system in the face of unknown threats.

[0011] To achieve the above object, the present invention proposes the following technical solutions: The present invention first discloses a network security protection method based on reinforcement learning, which is applicable to network security protection scenarios with multiple redundant execution units and specifically includes the following steps: In a preset heterogeneous executor pool provided on the defense platform, a heterogeneous executor combination is generated according to a preset redundancy to obtain a heterogeneous executor combination set; The heterogeneous executable combination set is updated according to the CVE vulnerability database to obtain a target heterogeneous executable combination set; When an abnormal threat event is identified, based on heuristic rules, calling the heterogeneous executable combination in the target heterogeneous executable combination set to defend against the abnormal threat event and obtaining a defense result; A protection strategy for the abnormal threat event is generated based on the defense result and the EXP3 algorithm, and a protection action is executed according to the protection strategy.

[0012] Optionally, the step of updating the heterogeneous executable combination set according to the CVE vulnerability database to obtain a target heterogeneous executable combination set includes: Collect vulnerability information at a preset time, and match the vulnerability information with the attribute information of the heterogeneous executable to obtain a matching result; The heterogeneous executable combination set is updated according to the matching result and the CVE vulnerability database to obtain a target heterogeneous executable.

[0013] Optionally, when an abnormal threat event is identified, the step of calling a heterogeneous executable combination in the target heterogeneous executable combination set based on a heuristic rule to defend against the abnormal threat event and obtaining a defense result includes: When an abnormal threat event is identified, obtaining the abnormal type of the abnormal threat event and the scope of the abnormal threat event, wherein the scope of the abnormal threat event includes the number of heterogeneous executable combinations attacked by the abnormal threat event and identification information of the heterogeneous executable combinations attacked by the abnormal threat event; generating alarm information for the abnormal threat event according to a preset alarm rule based on the abnormal type and the number of heterogeneous execution body combinations attacked by the abnormal threat event, and sending the alarm information to a preset device; When the warning level in the alarm information is greater than or equal to a preset level, the target heterogeneous executable combination set is updated according to the identification information of the heterogeneous executable combination attacked by the abnormal threat event to obtain an updated target heterogeneous executable combination set; Based on the heuristic rule, the updated heterogeneous executable combination in the target heterogeneous executable combination set is called to defend against the abnormal threat event, and a defense result is obtained.

[0014] Optionally, the method further includes: Obtaining a protection result of executing a protection action according to the protection strategy, and quantifying the protection result according to a preset quantization rule to obtain a quantized value of the protection result under the protection strategy; The toughness value of the EXP3 algorithm is calculated according to the quantized value, and the EXP3 algorithm is updated according to the toughness value.

[0015] Optionally, the method further includes: determining a sensitivity level of each heterogeneous executor combination in the target heterogeneous executor combination set based on the attribute data of each heterogeneous executor combination in the target heterogeneous executor combination set and the abnormal type of the abnormal threat event, treating heterogeneous executor groups with a sensitivity level greater than or equal to a preset level as cold devices, and decommissioning the cold devices so that the cold devices are removed from the preset heterogeneous executor pool; The identity information of the cold device is obtained, and based on the identity information, a backup device corresponding to the cold device is called in the backup device library to be added to the preset heterogeneous executor pool. After the cold device is repaired, the repaired cold device is restored to the preset heterogeneous executor pool.

[0016] Optionally, the method further includes: Obtaining source information of the abnormal threat event, and obtaining historical attack behavior data based on the source information; An attacker behavior probability model is established based on the historical attack behavior data, the abnormal threat event is input into the attacker behavior probability model, a prediction result is obtained, and protection is performed based on the prediction result.

[0017] On the other hand, the present application provides a network security protection system based on reinforcement learning, the system comprising: A combination module is used to generate a heterogeneous executable combination according to a preset redundancy in a preset heterogeneous executable pool provided on the defense platform to obtain a heterogeneous executable combination set; An update module, configured to update the heterogeneous executable combination set according to a CVE vulnerability database to obtain a target heterogeneous executable combination set; A first defense module is configured to, when an abnormal threat event is identified, call a heterogeneous executable combination in the target heterogeneous executable combination set based on a heuristic rule to defend against the abnormal threat event and obtain a defense result; The second defense module is used to generate a protection strategy for the abnormal threat event based on the defense result and the EXP3 algorithm, and perform a protection action according to the protection strategy.

[0018] In a third aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the network security protection method based on reinforcement learning as described above is implemented.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the above-mentioned reinforcement learning-based network security protection method.

[0020] Beneficial effects: This application generates heterogeneous executor combinations according to preset redundancy within a preset heterogeneous executor pool on a defense platform, obtaining a heterogeneous executor combination set. This heterogeneous executor combination set is then updated based on the CVE vulnerability database to obtain a target heterogeneous executor combination set. When an abnormal threat event is identified, the system uses heuristic rules to invoke heterogeneous executor combinations within the target heterogeneous executor combination set to defend against the abnormal threat event and obtain a defense result. Based on the defense result and the EXP3 algorithm, a protection strategy for the abnormal threat event is generated and defense actions are executed according to the strategy. This innovative research focuses on the inherent security defense mechanisms of cyber-physical systems (CPSs). The security of a dynamic heterogeneous redundancy (DHR) architecture is modeled and quantitatively analyzed under a real-world vulnerability attack environment. By introducing the EXP3 online learning algorithm, the system continuously optimizes the execution unit combination strategy during the attack-defense game to respond to changing attack patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a network security protection method based on reinforcement learning in the present invention; Figure 2 This is an attack and defense game process architecture diagram in a network security protection method based on reinforcement learning in the present invention; Figure 3 is a state transition cumulative curve diagram of the default parameters in the present invention; Figure 4 It is a state transition cumulative curve diagram of the alternative parameter 1 in the present invention; Figure 5 It is a state transition cumulative curve diagram of the alternative parameter 1 in the present invention; Figure 6 It is a state transition cumulative curve diagram of the alternative parameter 2 in the present invention; Figure 7 Schematic diagram of the structure of a network security protection system based on reinforcement learning in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 9 It is a structural diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0023] The embodiments described in this application are only part of the embodiments of the present invention, not all of the embodiments. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without making creative work are all within the scope of protection of the present invention.

[0024] The present invention is based on the collaborative work of multiple key steps and is applicable to physical information systems in fields such as industrial control, intelligent transportation, and energy Internet that have extremely high security and reliability requirements. It can significantly improve the system's resilience and fault tolerance in the face of known and unknown attacks. Figure 1 and Figure 2 As shown, a network security protection method based on reinforcement learning is applicable to network security protection scenarios with multiple redundant execution units, and specifically includes the following steps: S110: Generate a heterogeneous executable combination according to a preset redundancy in a preset heterogeneous executable pool provided on the defense platform to obtain a heterogeneous executable combination set; For example, a three-layer defense system is constructed: the bottom layer consists of multiple functionally equivalent but structurally heterogeneous executables (e.g., different operating systems, hardware platforms, or protection modules); the middle layer consists of dynamic heterogeneous combinations of executables (Heterogeneous Executor Units, or HEUs), which actually process user requests or tasks; and the top layer consists of a decision-making and scheduling module, responsible for adjusting defense strategies based on feedback from lower layers and the overall security status of the system. During the initialization phase, vulnerability information is labeled for all executables and executable combinations, a vulnerability-executor mapping table is established, and the set of all schedulable combinations is determined.

[0025] For example, multiple functionally equivalent but heterogeneous executables (also known as "functional units" or "defense nodes") are deployed on a physical or virtual defense platform. These executables can be software / hardware units with different operating systems, hardware architectures, or security mechanisms. For example, the operating system layer can deploy different kernel versions such as Windows, Linux, and RTOS, while the hardware layer can use different CPU platforms or network interface chips. All executables are managed uniformly by the system, forming a "heterogeneous executable pool." Each executable is assigned a unique identification number (ID), and its software and hardware configuration information is recorded.

[0026] Based on actual business needs and security policies, all or part of the "heterogeneous executor combinations" (HEU or EMU) are automatically generated from the above executor pool according to the preset redundancy (n-out-of-m). Each HEU consists of several (such as 3 or 5) different executors to deal with single point failures or tolerate the compromise of some executors. All optional HEU combination lists are initialized in matrix or index table format, and the total number of combinations can be calculated using combinatorial mathematical formulas (such as is the total number of executables, and n is the number of executables in the combination).

[0027] A three-tiered architecture is employed for collaborative work: the bottom tier comprises a pool of heterogeneous executors (the actual data and task carriers), the middle tier comprises the HEU portfolio management and adjudication module (implementing redundancy and diversity judgment outputs), and the top tier comprises the adaptive decision-making and control module (selecting scheduling strategies and updating reinforcement learning weights). Upon system startup, each module establishes communication channels and feedback interfaces. For example, HEU portfolio execution results are uploaded to the middle tier adjudication unit in real time, while the decision-making layer periodically pulls statistical data and pushes the latest scheduling solutions.

[0028] Assign an initial weight to each HEU combination (e.g. all weights are set to 1). For reinforcement learning related parameters, the learning rate , exploration factors, etc., are initialized according to empirical formulas or scenario parameters. The current status of each individual within the HEU combination is initialized to "undamaged / healthy." Data structures (such as a ring buffer and real-time log tables) are also established to accelerate subsequent attack feedback processing. After system initialization is complete, the control module enters a standby or alert state, awaiting input from external business traffic and security events.

[0029] S120, updating the heterogeneous executable combination set according to the CVE vulnerability database to obtain a target heterogeneous executable combination set; In a possible implementation, the step of updating the heterogeneous executable combination set according to the CVE vulnerability database to obtain a target heterogeneous executable combination set includes: Collect vulnerability information at a preset time, and match the vulnerability information with the attribute information of the heterogeneous executable to obtain a matching result; The heterogeneous executable combination set is updated according to the matching result and the CVE vulnerability database to obtain a target heterogeneous executable.

[0030] For example, a dynamic vulnerability-executable mapping table is maintained by combining the currently known CVE vulnerability database for each executable. New vulnerability information collected is automatically compared with each executable's firmware, software, configuration, and other version information, updating the set of affected executables in real time. Executables identified as having high-risk vulnerabilities are marked as high-risk nodes by the system, influencing the weight of subsequent defense combinations.

[0031] S130: When an abnormal threat event is identified, based on heuristic rules, calling the heterogeneous executable combination in the target heterogeneous executable combination set to defend against the abnormal threat event, and obtaining a defense result; In a possible implementation, when an abnormal threat event is identified, the step of invoking a heterogeneous executable combination in the target heterogeneous executable combination set based on a heuristic rule to defend against the abnormal threat event and obtaining a defense result includes: When an abnormal threat event is identified, obtaining the abnormal type of the abnormal threat event and the scope of the abnormal threat event, wherein the scope of the abnormal threat event includes the number of heterogeneous executable combinations attacked by the abnormal threat event and identification information of the heterogeneous executable combinations attacked by the abnormal threat event; generating alarm information for the abnormal threat event according to a preset alarm rule based on the abnormal type and the number of heterogeneous execution body combinations attacked by the abnormal threat event, and sending the alarm information to a preset device; When the warning level in the alarm information is greater than or equal to a preset level, the target heterogeneous executable combination set is updated according to the identification information of the heterogeneous executable combination attacked by the abnormal threat event to obtain an updated target heterogeneous executable combination set; Based on the heuristic rule, the updated heterogeneous executable combination in the target heterogeneous executable combination set is called to defend against the abnormal threat event, and a defense result is obtained.

[0032] For example, heuristic rules are used to provide preliminary warnings for perceived anomalies. For example, if a high probability of attack or high-risk vulnerability exploitation is detected, a preliminary scheduling response is triggered, including prioritizing the use of unaffected HEU combinations and reducing the priority of vulnerable executors. Scheduling decisions are monitored in real time, with defense effectiveness and attack feedback quickly transmitted back to the decision-making module, forming a preliminary feedback loop.

[0033] For example, in the attack detection module, the system analyzes multi-dimensional security event information, including security logs, network traffic, and host behavior, generated by key nodes in real time. Through abnormal behavior detection, signature rule comparison, and correlation analysis based on attack knowledge bases (such as ATT&CK and CVE), it identifies abnormal system calls, abnormal access requests, signals of potential vulnerability exploitation, or abnormal fluctuations in the state of certain executables. Heuristic detection rules not only rely on a priori known threats but also take into account unknown and variant attacks. Using algorithms such as feature anomalies, statistical deviations, and short-term frequency mutations, they dynamically mark high-risk targets and attack paths.

[0034] When an abnormal threat event is detected, such as when the security event score or risk level exceeds a preset threshold, or when the cumulative number of abnormal behaviors on some executors exceeds a warning level, the system immediately generates a rapid alert. A graded response is implemented based on the type and scope of the anomaly. For example, if only a few executors are abnormal, the system issues a warning based on priority and records it. If the attack chain spans multiple critical executors, the system automatically upgrades the alert to a high-level alert. The results of these graded alerts are directly reported to the scheduling module, which then suspends or restricts task allocation for high-risk executors.

[0035] Unaffected Heterogeneous Execution Units (HEUs) are prioritized for critical tasks, while HEUs with high-risk vulnerabilities or those already compromised are downgraded to backup or removed from the network. If resources are sufficient, defense redundancy can be added in parallel; otherwise, traffic allocation ratios can be dynamically adjusted to achieve rapid attack surface migration and risk isolation. Furthermore, the scheduling module records the effectiveness of this round of scheduling in real time, monitoring feedback on defense successes and failures, which serves as the basis for subsequent adaptive learning (EXP3 algorithm) weight adjustments.

[0036] After the scheduling response operation is completed, the real-time defense performance of the affected executors and the newly scheduled HEU is tracked. If the attack event is alleviated or the abnormal state is eliminated (for example, logs, calls, and resource anomalies return to normal), the system automatically lowers the warning level and dynamically restores the replaced or isolated executor combination. If the review detects scheduling failure, new attack spread, or continuous feedback failure, it automatically triggers lower-level resource protection or reports to the top-level decision-making layer to adjust reinforcement learning parameters. All feedback and scheduling information is continuously archived, forming a complete feedback loop.

[0037] S140 : Generate a protection strategy for the abnormal threat event based on the defense result and the EXP3 algorithm, and execute a protection action according to the protection strategy.

[0038] For example, within each scheduling cycle, the top-level decision module employs a multi-armed bandit reinforcement learning algorithm, such as EXP3, to assign weights to all available HEU combinations. Each time a HEU combination is selected and deployed for defense, its weight is dynamically adjusted based on its success or failure in resisting the attack. Successful defenses increase the probability of subsequent selection, while failures decrease it. The reinforcement learning algorithm continuously and adaptively selects the optimal HEU combination, enabling dynamic adaptation to unknown and evolving attacks. Furthermore, exploration strategies ensure that new combinations have a chance of being selected, thereby continuously improving overall defense capabilities.

[0039] For example, we first enumerate all candidate heterogeneous execution unit combinations (HEUs), each serving as an "arm" in reinforcement learning. We assign an initial weight to each HEU combination, and all combinations start with the same weight, indicating that each combination has an equal probability of being selected when its actual effect is unknown.

[0040] In each defense scheduling cycle, the system calculates the selection probability of each combination based on the weights of all current HEUs using the following formula of the EXP3 algorithm:

[0041] in, For the The probability of a HEU combination being selected, is its current weight, is the number of candidate combinations, This is an exploration parameter used to balance exploring unknowns with leveraging historically successful combinations. This ensures that all combinations have a chance of being selected as the current defense array, preventing defense blind spots caused by only a few combinations.

[0042] A HEU combination is randomly selected based on the above probability and becomes the current defense unit. It then takes on a round or batch of actual business flows or simulated attack flows. After completing this round of defense, the success or failure of this combination in resisting the attack is collected (e.g., whether it was breached; the defense status, which can be categorized as binary or multi-valued signals such as complete success, partial defense, or breach).

[0043] Based on the collected feedback, the system updates the weights of the HEU combinations participating in the defense. For EXP3, the weight update rule is:

[0044]

[0045] in, is the learning rate, Combination for the current round Reward value (different defense effects can set different reward points, such as full defense 1, partial defense 0.5, breakthrough 0), This is the feedback for probability normalization. The updated weights can increase the probability of high-performing combinations being selected in the future, thereby adaptively evolving to a safer scheduling strategy.

[0046] The scheduling strategy generation process is cyclical within each scheduling cycle. In long-term online games, the EXP3 algorithm theoretically minimizes cumulative losses (regret) in unknown or adversarial environments, gradually converging to the optimal or near-optimal defense combination distribution. This allows the system to adapt through continuous trial and error and feedback, continuously enhancing its overall resilience and security capabilities, even when attacker strategies shift or new vulnerabilities are exposed.

[0047] In a possible implementation, the method further includes: In order to deal with new attacks or sudden unknown threats in real environments, the system allows for the adjustment of reward signals and parameters (such as ) to make adaptive adjustments. For example, when a long period of large-scale failures is detected, the exploration parameters are automatically increased or the weight upper limit of some unexplored combinations is lowered to accelerate the convergence of the new model.

[0048] For example, when the number of consecutive failures exceeds the threshold N = 50 or the failure rate exceeds 80% within T = 100 steps, the following adjustments are automatically triggered: Increase the exploration parameter γ by Δγ = 0.1 (with an upper limit of γ ≤ 0.5) to enhance the exploration of the new arm; For arm i whose number of visits is less than the threshold V = 10, its weight w i The upper limit is lowered by 20%, that is, w i = min(w i , 0.8 × W max ); Increase the learning rate η by Δη = 0.01 (with an upper limit of η ≤ 0.1) to speed up weight updates; If the environment dynamics changes significantly, γ can be fine-tuned (e.g., γ ± 0.05, range [0.1, 0.5]) based on the failure mode evaluation.

[0049] The mathematical representation of adaptive adjustment is: Trigger conditions: Trigger condition = (N ≥ 50) ∨ (failure rate ≥ 0.8, T = 100) Improve the exploration parameter γ: γ new = min(γ current + 0.1, 0.5) Note: Increasing γ makes action selection more uniform and enhances random exploration.

[0050] Lowering the upper limit of the weight of the unexplored portfolio: w i = min(w i , 0.8 × W max ); Among them, W max The maximum value of all current arm weights is used to lower the upper limit of unexplored arm weights.

[0051] Adjust the learning rate η: η new = min(η current + 0.01, 0.1) Note: Increasing η accelerates weight updates to adapt to the new reward distribution.

[0052] Fine-tuning γ: γ new = max(min(γ current ± 0.05, 0.5), 0.1) Note: Fine-tune γ according to the dynamics of the environment and keep it in the range of [0.1, 0.5].

[0053] In a possible implementation, the method further includes: Obtaining a protection result of executing a protection action according to the protection strategy, and quantifying the protection result according to a preset quantization rule to obtain a quantized value of the protection result under the protection strategy; The toughness value of the EXP3 algorithm is calculated according to the quantized value, and the EXP3 algorithm is updated according to the toughness value.

[0054] For example, after each attack and defense cycle, the defense effect of the Heterogeneous Executing Unit (HEU) is collected and recorded in real time. For each attack attempt, it is determined whether the HEU is in one of the following three defense states, and the states are encoded and archived to obtain a defense state sequence: Majority fault defense (compromised state): More than half of the components in the HEU are attacked or fail, the system cannot provide normal services, and the status is marked as 0; Minority fault defense (partially damaged state): Some units in the HEU are damaged but not more than half, the system can degrade service but is not interrupted, and the status is marked as 1; Perfect Defense (fully available): All execution units are normal and undamaged, with the status mark being 2.

[0055] Based on the collected defense status sequence, the following three core resilience indicators are calculated to obtain the resilience measurement results: System availability and availability: This metric measures the total time the HEU is partially damaged or fully available (State 1 or 2) within a unit cycle, with the total number of attack cycles used as the denominator in the system availability calculation. This metric reflects the overall service availability of the defense system under sustained, intense attack.

[0056] Continuous Error Duration and Distribution: Detects and calculates statistics such as the longest continuous duration of the HEU in majority error defense (state 0), the average continuous error duration, and the error duration variance. This quantifies the speed and stability of the system's self-recovery from disaster states. If long periods of continuous errors are detected, the defense strategy is considered ineffective and a warning is generated.

[0057] State transition dynamics (transition probability and frequency): Using a sliding window statistical method, for each type of state transition (e.g., 0→1, 1→2, 2→0, etc.), the number of transitions is accumulated and the transition probability matrix within the window period is calculated. This metric can characterize the dynamic game between attack and defense, as well as the self-repair activity of the defense system.

[0058] The EXP3 algorithm is updated according to the toughness measurement results. Specifically: If the duration of consecutive errors exceeds the set threshold, or the actual availability is significantly lower than the expected target, the scheduling learning parameter fine-tuning (such as increasing the exploration rate, smoothing the weight distribution, etc.) will be automatically triggered to force the activation of more defense combination exploration and higher-frequency strategy switching to quickly escape the trapped state.

[0059] For example, the exploration rate ε is increased from the current value by 0.1 (the upper limit does not exceed 0.5), and the weight distribution is smoothed by applying entropy regularization to the action value Q(s, a) or the strategy probability distribution (the entropy weight coefficient is increased by 0.01), forcing the activation of more defense combination explorations and higher-frequency strategy switching to quickly escape from the trapped state.

[0060] If the state transition probability matrix shows that some defensive states dominate for a long time or the transition is rigid, the system will introduce de-weighting measures for high-weight combinations and introduce new combinations to improve the overall diversity and dynamic flexibility of the system.

[0061] Statistical results can also assist in implementing proactive alarms and external linkages, such as automatically notifying the security operations center to initiate manual intervention in the event of continuous breaches, or automatically isolating high-risk combinations and quickly performing hot standby recovery.

[0062] In a possible implementation, the method further includes: determining a sensitivity level of each heterogeneous executor combination in the target heterogeneous executor combination set based on the attribute data of each heterogeneous executor combination in the target heterogeneous executor combination set and the abnormal type of the abnormal threat event, treating heterogeneous executor groups with a sensitivity level greater than or equal to a preset level as cold devices, and decommissioning the cold devices so that the cold devices are removed from the preset heterogeneous executor pool; The identity information of the cold device is obtained, and based on the identity information, a backup device corresponding to the cold device is called in the backup device library to be added to the preset heterogeneous executor pool. After the cold device is repaired, the repaired cold device is restored to the preset heterogeneous executor pool.

[0063] For example, in response to detected advanced threat paths or attack chains, the system dynamically increases the sensitivity level of vulnerable HEU combinations or key executables. Specific measures include temporarily locking them out of scheduling, raising the threshold for selection, enhancing exploration penalties in the strategy, or reducing the scheduling frequency of related HEUs. This proactively avoids high-risk components and prevents attacks from concentrating on infiltrating weak links in the system.

[0064] For HEUs that have been unavailable for a period of time or whose attack success rate significantly exceeds a threshold, the system will initiate cold standby (temporarily taking them offline to observation / detection mode), initiate automatic diagnosis and repair processes, or directly decommission them (removing them from the scheduling pool to avoid affecting global resilience). Simultaneously, backup HEUs will be activated or restored to hot standby to ensure sufficient defense resources and continuous system availability.

[0065] Continuously perform automated testing on HEUs in cold standby or decommissioned states, including vulnerability patching, configuration refreshes, integrity verification, and security reinforcement. After repairs are complete, regression testing verifies that their defense capabilities have been effectively restored. Based on feedback, they are automatically re-added to the optional scheduling pool, enabling the recycling of defense resources and "self-healing" continuous recovery.

[0066] On the one hand, real-time performance data of recovery, repair, and switching operations are collected and re-input into the reinforcement learning decision model to optimize subsequent scheduling weights and parameters; on the other hand, HEU scheduling distribution and fault-tolerant safety margin are dynamically evaluated from a global perspective to ensure that no single executor or HEU experiences resource exhaustion and defense bottlenecks, ensuring the balance and resilience redundancy of the defense space.

[0067] In a possible implementation, the method further includes: Obtaining source information of the abnormal threat event, and obtaining historical attack behavior data based on the source information; Establish an attacker behavior probability model based on the historical attack behavior data, input the abnormal threat event into the attacker behavior probability model, obtain the prediction result, and perform protection based on the prediction result For example, for each identified attack event, the attack's "attack benefit" or risk level is quantified based on its corresponding CVE's CVSS risk score and actual impact. At the same time, historical attacker behavior is analyzed, including attack type distribution, penetration depth, and whether cascade attack strategies are employed, to establish a probabilistic model of attacker behavior. This model is used to infer the attacker's next possible intrusion path, improving the foresight of defense scheduling.

[0068] The specific experiments and analyses of the present invention are as follows: The experiments in this paper used multi-strategy comparison and reinforcement learning methods to evaluate the defense performance. The core parameter settings are shown in the table below, where the total number of executables is 10 and the number of unit EMU executables is set to 3. The probability of vulnerability existence is set to 0.4, the number of attack rounds is 50,000, and the sliding window size is 5,000 for data processing. During the EXP3 learning process, the learning rate According to the formula Calculations were performed to ensure policy adjustments in different scenarios. The experimental environment contained 70 known CVE vulnerabilities and nine known attack types. With the above configuration, the effectiveness of the defense strategy could be tested in real-world attack scenarios, and its performance under different vulnerabilities and attack modes could be evaluated.

[0069]

[0070] This paper sets up four types of defense strategies for comparative analysis to evaluate the performance and effectiveness of different defense methods in dealing with attacks. The strategies are: EXP3: Dynamic defense strategy based on perception and strategy selection. In each attack round, the defense system adjusts the selected defense strategy based on the previous attack results and feedback, thereby dynamically improving the defense effectiveness.

[0071] RAP: Randomized Perception Strategy. With this strategy, the defense system perceives changes in attack patterns or environmental conditions, gaining a certain degree of understanding of the attacker's behavioral characteristics and randomly selecting a defense strategy based on this understanding. While the perception process cannot achieve precise strategy optimization, it can leverage environmental information to improve defense adaptability.

[0072] Random: This strategy uses a completely random selection process. This strategy relies entirely on a random algorithm, without any optimization or learning process. During each attack round, the defense system randomly selects a server combination for defense. This strategy serves as a baseline for comparison with other, more intelligent defense strategies, helping to analyze their effectiveness and limitations.

[0073] Fixed: Fixed service combination strategy. Unlike the random strategy, the fixed strategy determines a fixed ECU at the beginning of the experiment, and the defense strategy does not change throughout the attack process.

[0074] As shown in the table below, under various parameter configurations, the EXP3-based strategy significantly outperforms the other three strategies in majority error defense, numerically reducing the next-best strategy by 9.35%, 28%, and 15.29%, respectively, demonstrating its superiority across various configurations. This demonstrates that the EXP3-based strategy consistently provides the best defense effectiveness across a wide range of scenarios. Furthermore, the EXP3-based strategy also performs well in terms of perfect defense performance, further demonstrating its excellent stability and ability to maintain effective defense capabilities under diverse attack modes and configurations. While relatively simple, the RAP-based and Random-based strategies are limited in their effectiveness due to a lack of in-depth decision optimization.

[0075]

[0076] The duration of consecutive errors reflects the speed at which the system transitions from anomaly to normal, demonstrating its resilience. As shown in the table below, under various parameter configurations, the EXP3-based strategy significantly outperforms the other three strategies in terms of maximum and average error lengths. This demonstrates that the EXP3-based strategy can restore the system to normal operation more quickly, demonstrating its superior resilience. The EXP3 algorithm also significantly lowers the error length variance metric than the other three strategies, indicating that the EXP3-based strategy provides a more stable defense and is less susceptible to extreme error fluctuations.

[0077]

[0078] like Figure 3-6 As shown in the following figures, the system state transition patterns under four different strategies (Exp3, RAP, Random, and Fixed) are shown. The state transition represents the cumulative number of transitions from one defense level to another. The horizontal axis is the number of attacks, and the vertical axis is the number of state transitions. The three types of state transitions include the transition from majority-error defense to minority-error defense (0 1) Most of the wrong defenses are transferred to perfect defenses (0 2) Minority error defense transfers to majority error defense (1 0) and perfect defense to majority fault defense (2 0). These state transitions represent the system's transition between defense failure (mostly wrong defense) and defense success (minority wrong defense, perfect defense), reflecting the stability of the system under different strategies.

[0079] from Figure 3-6As can be seen, overall, the Exp3 strategy has the slowest growth in the cumulative number of state transitions of all types, and the growth rate tends to be flat, which means that compared with other strategies, the Exp3 strategy exhibits higher stability. In contrast, the growth rate of state transitions of the RAP and Random strategies is similar, always higher than that of the Exp3 strategy, and shows no signs of convergence, which means that the RAP and Random strategies have difficulty achieving system stability. Due to the lack of a scheduling module, the state transitions of the Fixed strategy are completely dependent on the attacker's attack method. It can be found that although there is no fixed relationship between its growth rate and RAP and Random, its growth rate is always higher than that of the Exp3 strategy. Experimental results show that the Exp3 strategy has higher system stability than the comparison scheme and can reasonably schedule and achieve system stability under continuous attacks.

[0080] In summary, the Exp3 strategy demonstrates the best stability in the face of attacks of varying intensities and durations, making it an effective strategy for long-term defense. The RAP and Random strategies have certain error correction capabilities, but struggle to achieve stability as the attack intensity and duration increase. The Fixed strategy, due to its fixed heterogeneous lease numbers, lacks comprehensive error correction capabilities and is therefore unsuitable as a primary defense mechanism.

[0081] Compared to existing technologies, this paper innovatively investigates the inherent security defense mechanisms of cyber-physical systems (CPSs). It conducts attack-defense modeling and quantitative analysis of the security of a dynamic heterogeneous redundancy (DHR) architecture under a real-world vulnerability attack environment. The research demonstrates that systems with perception and decision-making capabilities exhibit greater resilience against cyberattacks, effectively suppressing both known and unknown threats while also offering advantages in dynamically adjusting defense strategies. By incorporating the EXP3 online learning algorithm, the system continuously optimizes the combined strategies of its execution units during the attack-defense game to address evolving attack patterns. Furthermore, the proposed three-dimensional resilience metric systematically characterizes the resilience of the DHR system based on key metrics such as the defense system's available time, the duration of consecutive errors, and the dynamic measurement of state transitions. Experimental validation demonstrates its scientific validity and effectiveness. The experimental results demonstrate that the dynamic heterogeneous redundancy (DHR) architecture offers significant security and resilience advantages over systems that achieve general perception and decision-making capabilities through heterogeneity. Defense strategies using the EXP3 online learning method can dynamically adjust defenses based on historical attack information, thereby reducing the probability of the system remaining in an abnormal state for extended periods after an attack and improving overall recovery capabilities. Further analysis shows that perception and decision-making capabilities significantly influence system resilience, and that different architectural parameter configurations (such as redundancy and decision-making mechanisms) exhibit significant differences in defense effectiveness.

[0082] On the other hand, Figure 7 As shown, the present application provides a network security protection system based on reinforcement learning, characterized in that the system includes: The combination module 201 is configured to generate a heterogeneous executable combination according to a preset redundancy in a preset heterogeneous executable pool provided on the defense platform to obtain a heterogeneous executable combination set; An updating module 202 is configured to update the heterogeneous executable combination set according to the CVE vulnerability database to obtain a target heterogeneous executable combination set; The first defense module 203 is configured to, when an abnormal threat event is identified, invoke the heterogeneous executable combination in the target heterogeneous executable combination set based on heuristic rules to defend against the abnormal threat event and obtain a defense result; The second defense module 204 is configured to generate a defense strategy for the abnormal threat event based on the defense result and the EXP3 algorithm, and execute a defense action according to the defense strategy.

[0083] In one possible implementation, Figure 8 As shown, an embodiment of the present application provides a terminal device 300, including: a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it is implemented to: generate a heterogeneous executable combination according to a preset redundancy in a preset heterogeneous executable pool provided on a defense platform to obtain a heterogeneous executable combination set; The heterogeneous executable combination set is updated according to the CVE vulnerability database to obtain a target heterogeneous executable combination set; When an abnormal threat event is identified, based on heuristic rules, calling the heterogeneous executable combination in the target heterogeneous executable combination set to defend against the abnormal threat event and obtaining a defense result; A protection strategy for the abnormal threat event is generated based on the defense result and the EXP3 algorithm, and a protection action is executed according to the protection strategy.

[0084] In one possible implementation, Figure 9 As shown, an embodiment of the present application provides a computer-readable storage medium 400 having a computer program 411 stored thereon. When the computer program 411 is executed by a processor, the computer program 411 is configured to: generate a heterogeneous executable combination according to a preset redundancy in a preset heterogeneous executable pool provided on a defense platform, thereby obtaining a heterogeneous executable combination set; The heterogeneous executable combination set is updated according to the CVE vulnerability database to obtain a target heterogeneous executable combination set; When an abnormal threat event is identified, based on heuristic rules, calling the heterogeneous executable combination in the target heterogeneous executable combination set to defend against the abnormal threat event and obtaining a defense result; A protection strategy for the abnormal threat event is generated based on the defense result and the EXP3 algorithm, and a protection action is executed according to the protection strategy.

[0085] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, removable hard drives, magnetic disks, or optical disks. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0088] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0089] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0090] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0091] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0092] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

[0093] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A network security protection method based on reinforcement learning, characterized in that: The following steps are involved: In a preset heterogeneous executor pool provided on the defense platform, a heterogeneous executor combination is generated according to a preset redundancy to obtain a heterogeneous executor combination set; The heterogeneous executable combination set is updated according to the CVE vulnerability database to obtain a target heterogeneous executable combination set; When an abnormal threat event is identified, based on heuristic rules, calling the heterogeneous executable combination in the target heterogeneous executable combination set to defend against the abnormal threat event and obtaining a defense result; A protection strategy for the abnormal threat event is generated based on the defense result and the EXP3 algorithm, and a protection action is executed according to the protection strategy.

2. A network security protection method based on reinforcement learning according to claim 1, characterized in that: The step of updating the heterogeneous executable combination set according to the CVE vulnerability database to obtain a target heterogeneous executable combination set includes: Collect vulnerability information at a preset time, and match the vulnerability information with the attribute information of the heterogeneous executable to obtain a matching result; The heterogeneous executable combination set is updated according to the matching result and the CVE vulnerability database to obtain a target heterogeneous executable.

3. A network security protection method based on reinforcement learning according to claim 1, characterized in that: The step of, when an abnormal threat event is identified, calling a heterogeneous executable combination in the target heterogeneous executable combination set based on a heuristic rule to defend against the abnormal threat event and obtaining a defense result includes: When an abnormal threat event is identified, obtaining the abnormal type of the abnormal threat event and the scope of the abnormal threat event, wherein the scope of the abnormal threat event includes the number of heterogeneous executable combinations attacked by the abnormal threat event and identification information of the heterogeneous executable combinations attacked by the abnormal threat event; Generate alarm information for the abnormal threat event according to a preset alarm rule based on the abnormal type and the number of heterogeneous execution body combinations attacked by the abnormal threat event, and send the alarm information to a preset device; When the warning level in the alarm information is greater than or equal to a preset level, the target heterogeneous executable combination set is updated according to the identification information of the heterogeneous executable combination attacked by the abnormal threat event to obtain an updated target heterogeneous executable combination set; Based on the heuristic rule, the updated heterogeneous executable combination in the target heterogeneous executable combination set is called to defend against the abnormal threat event, and a defense result is obtained.

4. A network security protection method based on reinforcement learning according to claim 1, characterized in that: The method further comprises: Obtaining a protection result of executing a protection action according to the protection strategy, and quantifying the protection result according to a preset quantization rule to obtain a quantized value of the protection result under the protection strategy; The toughness value of the EXP3 algorithm is calculated according to the quantized value, and the EXP3 algorithm is updated according to the toughness value.

5. The network security protection method based on reinforcement learning according to claim 3 is characterized in that: The method further comprises: determining a sensitivity level of each heterogeneous executor combination in the target heterogeneous executor combination set based on the attribute data of each heterogeneous executor combination in the target heterogeneous executor combination set and the abnormal type of the abnormal threat event, treating heterogeneous executor groups with a sensitivity level greater than or equal to a preset level as cold devices, and decommissioning the cold devices so that the cold devices are removed from the preset heterogeneous executor pool; The identity information of the cold device is obtained, and based on the identity information, a backup device corresponding to the cold device is called in the backup device library to be added to the preset heterogeneous executor pool. After the cold device is repaired, the repaired cold device is restored to the preset heterogeneous executor pool.

6. A network security protection method based on reinforcement learning according to claim 1, characterized in that: The method further comprises: Obtaining source information of the abnormal threat event, and obtaining historical attack behavior data based on the source information; An attacker behavior probability model is established based on the historical attack behavior data, the abnormal threat event is input into the attacker behavior probability model, a prediction result is obtained, and protection is performed based on the prediction result.

7. A network security protection system based on reinforcement learning, characterized in that: The system comprises: A combination module is used to generate a heterogeneous executable combination according to a preset redundancy in a preset heterogeneous executable pool provided on the defense platform to obtain a heterogeneous executable combination set; An update module, configured to update the heterogeneous executable combination set according to a CVE vulnerability database to obtain a target heterogeneous executable combination set; A first defense module is configured to, when an abnormal threat event is identified, call a heterogeneous executable combination in the target heterogeneous executable combination set based on a heuristic rule to defend against the abnormal threat event and obtain a defense result; The second defense module is used to generate a protection strategy for the abnormal threat event based on the defense result and the EXP3 algorithm, and perform a protection action according to the protection strategy.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the network security protection method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the network security protection method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

Images