A reinforcement learning-based method for selecting and distributing collaborative defense strategies for heterogeneous security devices.

By using reinforcement learning to perform unified modeling and collaborative defense of heterogeneous security devices, the problem of policy duplication and conflict between heterogeneous security devices is solved, and efficient collaborative defense and policy execution are achieved.

CN123122933APending Publication Date: 2026-11-17GUANGZHOU UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611342472.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies make it difficult to achieve unified modeling and collaborative defense among heterogeneous security devices, making it difficult for operations and maintenance personnel to judge the coverage of policies from a global perspective, resulting in problems such as duplicate configuration, rule conflicts, and uncoordinated policy execution.

Method used

A reinforcement learning approach is used to uniformly model the native rules of heterogeneous security devices, construct an attack-defense state and candidate action space, select devices and their policy actions to participate in collaborative defense, and achieve adaptive collaborative defense of heterogeneous security devices through policy verification, native rule transformation and feedback update.

Benefits of technology

It enables adaptive and collaborative defense among heterogeneous security devices, improves the executability and security of policy selection, reduces the impact of policy execution on normal business operations, and forms a closed-loop mechanism for policy selection, verification, distribution, monitoring and feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123122933A_ABST
    Figure CN123122933A_ABST
Patent Text Reader

Abstract

This invention discloses a method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning. The method includes: accessing multiple types of heterogeneous security devices, collecting native rules from these devices and parsing and mapping them into a unified policy object to achieve a standardized representation of heterogeneous security policies; upon receiving alarm events, fusing threat intelligence, network assets, device handling capabilities, and existing policies to construct an attack-defense state, generating candidate device-policy actions, and excluding inexecutable, conflicting, and high-risk actions through dynamic security action masks; within a constrained action space, selecting collaborative defense strategies through a reinforcement learning model, verifying the strategies, and then converting them into native device rules for collaborative distribution; monitoring the execution effect, calculating feedback rewards, and updating the reinforcement learning model to form a complete closed loop of strategy selection, execution, and optimization. This invention can effectively support collaborative defense decisions for heterogeneous security devices, improving the adaptability and reliability of security strategies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the technical field of automated management of network security operation and maintenance strategies, specifically involving a method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning. Background Technology

[0002] As digital transformation continues to advance, the complexity of enterprise network environments is constantly increasing, and cybersecurity threats are showing a trend towards diversification, concealment, and high frequency. To ensure network and business security, enterprises typically deploy heterogeneous security devices from different vendors and of different types, with these devices jointly undertaking security functions such as network access control, attack detection, attack blocking, and security auditing.

[0003] However, different security devices differ in rule syntax, field definitions, response actions, and management interfaces. Security policies for various devices are typically configured separately by operations and maintenance personnel, lacking a unified representation and management method. This situation makes it difficult for operations and maintenance personnel to comprehensively assess the coverage of existing security policies against current attacks from a global perspective, and also makes it difficult to efficiently identify anomalies such as duplicate configurations and rule conflicts between different device policies. Furthermore, in actual operation, each security device typically performs detection, alerting, auditing, or blocking operations independently based on its own alarm information and local rules. When a device detects an attack, the existing system struggles to combine real-time attack-defense status to select a suitable collaborative response plan from multiple heterogeneous security devices and their supported policy actions. In addition, directly generating and issuing new security policies based on alarm information can easily lead to problems such as target devices not supporting corresponding response actions, conflicts between new and existing policies, repeated modifications to existing rules, and disruption of normal business operations.

[0004] Existing technologies related to security policy modeling, dynamic optimization, and automated execution can be mainly divided into the following three categories:

[0005] The first category comprises security policy management methods based on rule modeling and policy anomaly analysis. These methods typically formally represent rule fields, matching conditions, actions, and execution order within security devices. By analyzing the inclusion, intersection, and action logic relationships between rules, they identify issues such as duplication, obfuscation, conflicts, and configuration anomalies within the policy. For example, the firewall policy modeling and management method proposed by AL-SHAER et al. can automatically detect rule conflicts and potential configuration problems by analyzing the logical relationships between firewall filtering rules, and perform anomaly detection during rule addition, deletion, or modification to reduce security risks caused by rule configuration errors. However, this type of solution primarily addresses static analysis scenarios of firewall policies and has not yet solved the problem of unified rule representation across different types of security devices, nor can it complete device selection and corresponding policy action matching under multi-device collaboration based on real-time attack-defense status.

[0006] The second category comprises adaptive optimization methods for security policies based on machine learning or reinforcement learning. These methods typically treat network traffic, alarm information, or device operating status as the environment state, and operations such as adding rules, modifying rules, blocking traffic, and restricting access as actions. They combine attack detection results with network operating status to calculate reward values, enabling the policy selection model to autonomously adjust and optimize security policies through interactive learning with the environment. For example, the AI-driven dynamic firewall optimization framework proposed by AHMAD et al. models the firewall rule adjustment process as a Markov decision process. A reinforcement learning agent perceives the network state, and a hybrid model composed of long short-term memory networks and convolutional neural networks detects abnormal traffic, dynamically adjusting the firewall configuration based on the detection results. This scheme verifies the feasibility of reinforcement learning in dynamic security policy adjustment scenarios; however, its action space is only built around firewall rule operations, primarily targeting single-type firewall devices. It does not establish a unified device-policy action representation system for heterogeneous security devices, nor does it address issues such as compatibility verification between multiple device actions, policy conflict avoidance, and business risk constraints.

[0007] The third category comprises rule optimization and automatic policy execution methods based on large language models. These methods leverage the semantic understanding capabilities of large language models to parse security requirements, device rules, or operational context information in natural language form, thereby generating new rule content, policy modification suggestions, or executable interface call instructions. For example, the large language model adaptive rule optimization method proposed by Ishimizu et al. uses a monitoring-analysis-planning-execution-knowledge base loop framework. The large language model analyzes existing rules and system operational information, identifies rule shortcomings, and plans new adaptive rules. This approach demonstrates the auxiliary value of large language models in rule understanding and optimization scenarios, but it primarily focuses on rule design for adaptive application systems and does not address the issues of rule adaptation to heterogeneous security devices and multi-device collaborative policy selection. Another example is the large language model attack mitigation policy automatic execution framework proposed by SAURA et al., which uses retrieval enhancement generation technology to obtain relevant tool and application programming interface information, decomposes the natural language-described attack mitigation policies into multiple execution tasks, and further converts them into executable interface calls. This solution can narrow the semantic gap between high-level security policies and low-level execution interfaces, but its core focus is on the executable transformation of established security policies. It does not solve the problem of how to select collaborative defense devices and corresponding policy actions based on real-time attack-defense status. At the same time, it lacks a dynamic security constraint mechanism for candidate actions and does not establish a reinforcement learning feedback update path based on actual execution results.

[0008] In summary, the existing rule systems of heterogeneous security devices lack unified modeling, making it difficult to uniformly determine the coverage of existing policies against current attacks, and making it difficult to avoid duplication and conflict during cross-device policy adjustments. When heterogeneous security devices generate threat alerts, the lack of a collaborative policy selection mechanism based on the current attack-defense status prevents effective linkage between security devices, resulting in attacks not being dealt with in a timely manner. Therefore, providing an adaptive collaborative defense mechanism between heterogeneous security devices to address the above shortcomings is an urgent issue to be resolved. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning. This method unifies the original rules of existing heterogeneous security devices, constructs an attack-defense state and a candidate action space subject to security constraints, uses reinforcement learning to select security devices and their corresponding strategy actions to participate in collaborative defense, and achieves closed-loop updates through strategy verification, original rule conversion, collaborative distribution, and execution feedback, thereby realizing adaptive collaborative defense among heterogeneous security devices.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning, comprising the following steps:

[0012] Collect the native rules of heterogeneous security devices, and transform the native rules of the devices into a unified policy object according to a preset rule parsing template. Each security device in the heterogeneous security devices has a unique device identifier.

[0013] When an alarm event is received, the alarm event is associated with threat intelligence, network asset information and device defense context to construct the attack-defense state corresponding to the current alarm event. The attack-defense state is used as input to generate candidate device-policy actions from the preset policy action template library. The corresponding dynamic security action mask is generated based on the candidate device-policy actions, and the set of executable candidate device-policy actions is obtained based on the dynamic security action mask.

[0014] Input the attack-defense status and the set of executable candidate devices-policy actions. Use a reinforcement learning policy selection model to select security devices participating in the linkage and their corresponding policy actions. Obtain executable device-policy actions and form a structured collaborative candidate policy set. Validate the structured collaborative candidate policy set. Convert the validated structured collaborative candidate policy set into device native rules that the target security devices can recognize. Distribute the device native rules to one or more corresponding target security devices and record the policy distribution results.

[0015] The target security device loads and executes the issued native device rules, monitors the execution results of the target security device after executing the native device rules, calculates feedback rewards based on the execution results and generates policy selection experience, and uses the policy selection experience to update the reinforcement learning policy selection model.

[0016] As a preferred technical solution, the method of collecting the native rules of heterogeneous security devices and converting them into a unified policy object according to a preset rule parsing template is as follows:

[0017] Based on the access configuration records of heterogeneous security devices, the native rules currently configured for each security device are obtained through the device management interface, rule file reading interface, or other preset collection methods. The access configuration records include device identifier, device type, manufacturer type, management address, rule format, rule acquisition method, rule parsing template, target device rule template, and policy distribution interface information.

[0018] For security devices with different deployment methods, the data collection component is invoked to read the device's native rules. At the same time, the changed native rules of the device are identified by rule version, update time, or file hash value.

[0019] The access configuration records, collection time, and rule file change status are associated and saved with the device's native rules to obtain the device's native rule set;

[0020] The system parses the set of native rules for each device. Based on the device type, manufacturer type, and rule format associated with each device's native rules, it selects the appropriate rule parsing template, parses the fields of the set of native rules for each device according to the rule parsing template, and constructs a unified policy object. The rule parsing template includes rule segmentation rules, field location rules, field type conversion rules, and field value mapping rules.

[0021] As a preferred technical solution, when an alarm event is received, the alarm event is associated with threat intelligence, network asset information, and device defense context to construct the attack-defense state corresponding to the current alarm event, specifically as follows:

[0022] The system receives alarm data generated by heterogeneous security devices or security operation platforms, extracts alarm information from the alarm data, and forms alarm events, including alarm identifier, source device, attack source, attack target, communication protocol, attack type, alarm level, alarm time, and attack characteristics. For multiple alarm messages with the same attack conditions or similar attack characteristics within a preset time window, the alarm information is aggregated into the same alarm event, and the number of alarm occurrences and the highest alarm level are recorded. The attack conditions include attack source, attack target, and attack type.

[0023] The alarm events are used as an index to associate threat intelligence, network asset information, and device defense context. The device defense context includes security device protection relationships, device handling capabilities, device operating status, and unified policy objects. The threat intelligence is used to determine the malice level and intelligence credibility of the attack source. The network asset information is used to determine the business attributes and importance level of the target asset. The device defense context is used to determine the security devices that can participate in the current attack handling and their existing device defense capabilities.

[0024] Based on the matching relationship between the unified policy object and the current alarm event, the policy coverage status of each security device for the current attack is determined. The policy coverage status includes no policy coverage, detection coverage, alarm coverage, audit coverage, blocking coverage, and failure coverage.

[0025] By using alarm events, threat intelligence, and network asset information as the current attack risk status, and the unified policy object, device defense context, and policy coverage status as the current defense capability status, the attack-defense status at the current moment is constructed.

[0026] As a preferred technical solution, the preset policy action template library is classified according to the type of security device, and is used to record the policy operation type, matching field, handling action, scope of application and activation method supported by different types of security devices;

[0027] Each candidate device-policy action consists of the target security device and its corresponding policy operation, as shown in the following formula:

[0028] ,

[0029] in, Indicates the target security equipment. Indicates the operation type, including adding a policy, modifying a policy, or enabling an existing policy. Indicates the policy matching conditions. This indicates the actions taken, including alerting, auditing, limiting speed, blocking, or denying access. Indicates the scope of the strategy. Indicates the duration for which the policy is in effect;

[0030] Multiple candidate device-policy actions constitute the candidate device-policy action set at the current moment.

[0031] As a preferred technical solution, the step of generating a corresponding dynamic security action mask based on the candidate device-policy action specifically includes:

[0032] For each candidate device-policy action, a security action mask is generated based on device adaptation constraints, policy consistency constraints, and business security constraints. For each candidate device-policy action, its security action mask is... Represented as:

[0033] ,

[0034] in, This indicates the result of the device adaptation constraint judgment. This indicates the result of the judgment regarding the policy consistency constraint. This indicates the result of the judgment on business security constraints. This represents the logical AND operation;

[0035] The set of executable candidate device-policy actions is obtained based on the mask value. Candidate device-policy actions with a mask value of 0 are masked in the current candidate action space, and the corresponding constraint type and masking reason are recorded.

[0036] During the sequential action selection process, after each candidate device-policy action is selected, the above three types of constraints are re-evaluated based on the virtual updated attack-defense status, and the mask values ​​of the remaining candidate device-policy actions are dynamically updated.

[0037] As a preferred technical solution, the reinforcement learning strategy selection model adopts a decision-making method based on the state-action value function to calculate the action value of each executable candidate device-strategy action in the current attack-defense state; and selects the executable candidate device-strategy action with the highest action value from the set of executable candidate device-strategy actions, that is, obtains the executable device-strategy action;

[0038] The input attack-defense status and executable candidate device-policy action set are used to select participating security devices and corresponding policy actions using a reinforcement learning policy selection model, thereby obtaining executable device-policy actions and forming a structured collaborative candidate policy set. Specifically:

[0039] Using a reinforcement learning policy selection model, the executable candidate device-policy action with the highest action value is selected from the executable candidate device-policy action set to obtain the executable device-policy action. The executable device-policy action is added to the set to obtain a structured collaborative candidate policy set. Based on the selected executable device-policy action, the existing policy coverage state and device defense capabilities are virtually updated to obtain the attack-defense state for the next round.

[0040] As a preferred technical solution, the executable candidate device-strategy action set also includes an end action. If the reinforcement learning strategy selection model selects an end action, the updated strategy coverage state meets the preset handling requirements, the number of selected actions reaches the preset upper limit, or there are no other executable candidate actions, the selection process is terminated. If the existing strategy can already meet the handling requirements of the current attack, the reinforcement learning strategy selection model directly selects an end action in the first round and stops generating new strategy change actions.

[0041] As a preferred technical solution, the step of verifying the structured collaborative candidate strategy set, converting the verified structured collaborative candidate strategy set into device native rules that can be recognized by the target security device, and distributing the device native rules to one or more corresponding target security devices, specifically involves:

[0042] Perform the following risk checks on the structured collaborative candidate strategy set in sequence, including:

[0043] Integrity verification: Check whether each executable device-policy action in the structured collaborative candidate policy set has the necessary information, and whether the field types and value ranges conform to the unified policy object specification; the necessary information includes target device, policy operation type, matching conditions, disposal action, scope of effect and duration of effect;

[0044] Executability verification: Based on the current operating status and handling capabilities of the target device, confirm that the target device is still available and supports the corresponding matching fields, handling actions, and activation methods;

[0045] Combinatorial consistency verification: Determine whether there are duplicate, inclusion, overriding or conflicting relationships between multiple executable device-policy actions, or between the actions and existing policies, and obtain the verification result; the existing policies are the device native rules that were configured on each target security device before the current alarm event occurred and were parsed into a unified policy object.

[0046] Business risk verification: Based on the policy's scope of action, the target asset's importance level, affected ports, normal business traffic range, and management channel information, determine whether the executable device-policy action may cause critical business interruption, large-scale false blocking, or management channel unavailability; if the expected business risk exceeds the preset risk threshold, the expected business is marked as a high-risk policy.

[0047] The verification result is represented as:

[0048] ,

[0049] in, Indicates the integrity verification result. This indicates the executability check result. This indicates the result of the combinational consistency check. This indicates the results of the business risk verification.

[0050] When the verification result is true, the structured collaborative candidate strategy set passes the verification; otherwise, the action that fails the verification and its reason are recorded, the corresponding executable device-strategy action is deleted, and the reinforcement learning strategy selection model is used to continue the selection or the case is transferred to manual review.

[0051] The structured collaborative candidate policy set that has passed the verification is used as input. The access configuration record of the device is queried according to the device identifier to obtain the corresponding target device rule template and policy distribution interface information. The target device rule template is used to describe the rule syntax, field name, field order, action keyword, value range and necessary parameters supported by the target security device.

[0052] Based on the unified fields in the executable device-policy action and the target device rule template, generate native device rules that the target security device can recognize. For the first... An executable device-policy action, whose rule transformation process is represented as follows:

[0053] ,

[0054] in, This indicates an executable device-policy action that has passed verification. Indicates target security equipment The corresponding target device rule template, Represents a rule transformation function. This represents the native device rules of the target device obtained after the conversion;

[0055] The policy is sent to the corresponding security device through the policy distribution interface of the target device. When multiple security devices are involved, the policy is distributed according to the action order or device dependency relationship in the structured collaborative candidate policy set. The independent executable devices and policy actions are executed in parallel.

[0056] As a preferred technical solution, the rule conversion function adopts a large language model, which will complete the conversion of the unified field to the device's native rule syntax;

[0057] Once the native rules for the target security device are generated, syntax validation is performed based on the target device rule template to check whether the keywords, number of fields, field types, parameter formats, and necessary fields in the rules meet the requirements of the target device.

[0058] For native device rules that fail syntax validation, the error fields are recorded and the rule conversion is re-executed. If the validation still fails after the preset number of retries, the rule is transferred to manual review and automatic distribution is stopped.

[0059] For native device rules that pass syntax validation, execute new policies, modify policies, or enable existing policies according to the policy operation type, and send them to the corresponding security devices through the policy distribution interface of the target device.

[0060] Record the target device, policy operation type, device response result, and distribution status corresponding to the native rules of each device; if the interface call fails, the device refuses to execute, or the response times out, stop subsequent distribution operations that are dependent on the failed action, and record the reason for the failure.

[0061] As a preferred technical solution, the target security device loads and executes the issued native device rules, monitors the execution results of the target security device after executing the native device rules, calculates feedback rewards based on the execution results and generates policy selection experience, and updates the reinforcement learning policy selection model using the policy selection experience, specifically as follows:

[0062] If the distribution is successful and the target device is operating normally, perform the following steps:

[0063] Within a preset observation period, monitor the policy hit rate, attack blocking status, device operation status, and normal business operation status of the target security device, and confirm whether the issued device native rules have been loaded and are enabled through the device query interface.

[0064] Compare the number of alarms, attack traffic, and policy hit rate before and after policy execution to determine whether the structured collaborative candidate policy set has achieved the expected handling effect. At the same time, judge the impact of the policy on normal business based on the normal request failure rate, business response time, critical service availability, and management channel connectivity status.

[0065] If attack alerts continue to occur, the policy does not produce the expected hit, normal business is abnormal, the scope of false blocking exceeds the preset threshold, the device is operating abnormally, or there is a policy conflict between multiple devices, then this execution will be marked as abnormal, and the newly added rule will be revoked, the modified rule will be restored, or the original rule will be reactivated according to the original rule version of the device saved before the issuance.

[0066] The feedback reward for this collaborative strategy is calculated based on the attack handling effect, the degree of policy coverage improvement, the business impact, the cost of policy change, and the execution anomaly. A positive reward is obtained when the attack is successfully handled and does not affect normal business, and a negative reward is obtained when the attack continues, business anomalies occur, or a rollback is triggered.

[0067] Save the attack-defense state before strategy selection, the set of structured collaborative candidate strategies, the feedback reward, and the defense state after strategy execution as a strategy selection experience.

[0068] According to the action selection order, the current structured collaborative candidate strategy set is divided into the selected device-strategy action in each round. The virtual attack-defense state before each round of selection, the selected device-strategy action, the feedback reward, and the virtual attack-defense state after selection are formed into action-level strategy selection experience. The feedback reward is distributed to the selected executable device-strategy action in each round according to a preset discount method, which is used to update the state-action value function in the reinforcement learning strategy selection model.

[0069] The action-level policy selection experience is written into the historical experience database. When the number of experiences reaches the preset update condition, the newly added experience is used to train the reinforcement learning policy selection model offline or update it periodically. The updated model replaces the current model after it has been verified.

[0070] For cases where the distribution fails, the device response times out, or the rule does not actually take effect, the execution is marked as abnormal and a negative reward is given to form corresponding strategy selection experience. At the same time, the corresponding target security device, executable device-policy action, and failure reason are recorded.

[0071] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0072] (1) This invention can realize a unified representation of the native rules of heterogeneous security devices. It extracts fields, converts types and maps field values ​​for the native rules of heterogeneous security devices, and stores the vendor-specific parameters that cannot be uniformly mapped in the extended fields to form a unified policy object with a consistent structure. Therefore, it can eliminate the differences in rule syntax, field names and action representations between different devices, and provide a unified data foundation for judging the coverage status of existing policies and selecting subsequent cross-device collaborative policies.

[0073] (2) This invention can improve the executability and security of reinforcement learning collaborative strategy selection. Because this invention constructs an attack-defense state based on alarm events, threat intelligence, network assets and device defense context, and generates candidate device-policy actions based on device handling capabilities; at the same time, it uses device adaptation constraints, policy consistency constraints and business security constraints to generate dynamic security action masks to shield candidate actions that are not supported by the device, duplicate or conflict with existing policies, or have excessive business risks. Therefore, the reinforcement learning strategy selection model can make decisions within a constrained action space, reducing the possibility of unexecutable or inappropriate policies being selected.

[0074] (3) The present invention can realize adaptive collaborative processing of multiple heterogeneous security devices. The present invention adopts a sequential action selection method. After the reinforcement learning strategy selection model selects a device-strategy action, it virtually updates the existing strategy coverage state and defense capability state, and recalculates the dynamic security action mask, and continues to select subsequent actions until the preset termination condition is met. Therefore, it is not necessary to enumerate all device and strategy action combinations in advance to form a structured collaborative candidate strategy set jointly executed by one or more heterogeneous security devices, and dynamically adjust the subsequent collaborative scheme according to the selected actions.

[0075] (4) This invention can improve the reliability of the collaborative strategy issuance and execution process. Before issuance, this invention verifies the completeness, executability, combination consistency and business risk of the structured collaborative candidate strategy set, converts the verified strategy actions into native rules supported by the target device and issues them collaboratively; after issuance, it continuously monitors the effectiveness of the strategy, the effect of attack handling and the normal business operation status, restores the original device rule version before issuance when an anomaly occurs, and calculates feedback rewards based on the execution results, saves strategy selection experience and updates the reinforcement learning strategy selection model. Therefore, it can form a closed loop of strategy selection, verification, issuance, monitoring, rollback and feedback update, reducing the risk of strategy execution failure or affecting normal business. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a flowchart illustrating the method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning, as described in an embodiment of the present invention.

[0078] Figure 2 This is an architecture diagram of the intelligent optimization platform for collaborative defense strategies according to an embodiment of the present invention;

[0079] Figure 3 This is a flowchart illustrating the reinforcement learning sequential collaborative strategy selection process according to an embodiment of the present invention. Figure 3 Part (a) shows the first Round attack—defense status and executable candidate devices—strategy action input, status—action value calculation, maximum action value selection and status virtual update process; Figure 3 Part (b) shows the selected equipment and strategy actions in each round, forming a structured set of collaborative candidate strategies in the order of selection. The process. Detailed Implementation

[0080] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0081] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0082] The terms used in this application are explained as follows:

[0083] Device native rules: These are security rules that security devices can directly recognize and execute, written using the device's own syntax.

[0084] Unified policy object: refers to a data object formed by parsing and standardizing fields such as address, port, protocol, matching conditions and action in the native rules of different devices.

[0085] Dynamic security action mask: refers to the constraint result of marking candidate devices and policy actions as optional or non-optional based on device adaptability, policy consistency and business risks.

[0086] Virtual update: refers to updating the policy coverage status and defense capability status based on the selected device and policy action during the sequential action selection process, so that it can be used for subsequent action selection, but without directly modifying the original device rules of the real security device.

[0087] This embodiment provides a method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning. The process is shown in sequence as follows: collection of native rules and construction of unified policy objects (S101-S102), construction of attack-defense status and dynamic security action mask (S103-S104), selection of collaborative defense strategies based on reinforcement learning (S105), and processing steps such as candidate strategy verification, conversion of native rules and collaborative distribution, execution monitoring and feedback update (S106-S108).

[0088] Please see Figure 1 The above method specifically includes the following steps:

[0089] Step S101: Connect to heterogeneous security devices and collect native rules from the devices. The collaborative defense strategy intelligent optimization platform (hereinafter referred to as the platform) connects to heterogeneous security devices such as firewalls, web application firewalls, intrusion detection systems, intrusion prevention systems, and database auditing systems, and assigns a unique device identifier to each security device. The platform establishes a device access configuration record for each security device. The device access configuration record includes at least the device identifier, device type, vendor type, management address, rule format, rule acquisition method, rule parsing template, target device rule template, and policy distribution interface information.

[0090] The overall architecture of the collaborative defense strategy intelligent optimization platform used in this embodiment is as follows: Figure 2As shown, the platform uses heterogeneous security devices, alarm events, threat intelligence, and network asset information as data sources. Logically, it includes an access and unified modeling layer, a state awareness and action generation layer, a collaborative decision-making layer, a policy execution layer, and a feedback update layer. The access and unified modeling layer includes a device access and native rule collection module and a unified policy object construction module; the state awareness and action generation layer includes an attack-defense state construction module and a candidate device-policy action and dynamic security action mask generation module; the collaborative decision-making layer includes a reinforcement learning collaborative policy selection module; the policy execution layer includes a structured collaborative candidate policy set verification module and a target device native rule conversion and collaborative distribution module; and the feedback update layer includes an execution effect monitoring and feedback update module. Through these modules, the platform sequentially completes the unified modeling of native rules for heterogeneous security devices, attack-defense state construction, collaborative policy selection, native rule conversion and distribution, and execution feedback updates. It then distributes the generated target device native rules to execution objects such as firewalls, web application firewalls, intrusion detection / intrusion prevention systems, and database auditing systems.

[0091] Based on the device access configuration records, the platform obtains the currently configured native rules for each security device through the device management interface, rule file reading interface, or other preset collection methods. For security devices with different deployment methods, the platform calls the corresponding rule collection component to complete the rule reading.

[0092] In this embodiment, the data collection component is a security device native rule collection script written in Python. The platform pre-configures the target security device's management IP address, SSH port, username, and password, and records the corresponding native rule file type and storage path. When executing the collection task, the script connects to the security device in the topology via SSH based on the target device's management IP address and authentication information, locates the rule file according to the preset storage path, and reads and collects the corresponding file according to its rule file type. After collection, the rule file, device identifier, file type, original storage path, and collection time are transmitted back to the collaborative defense strategy intelligent optimization platform via the SCP protocol.

[0093] Meanwhile, changed native rules for devices can be identified by rule version, update time, or file hash value to avoid duplicate data collection. For example, after collecting a Suricata rule from an intrusion detection system identified as IDS-001, the following record is generated:

[0094] {

[0095] "raw_data_id": "RAW-RULE-0001",

[0096] "device_id": "IDS-001",

[0097] "device_type": "Intrusion Detection System",

[0098] "rule_file_type": "Suricata .rules rules file",

[0099] "collect_time": "2026-01-01 10:00:00",

[0100] "file_change_status": "Added",

[0101] "raw_content": "alert http any any -> any any (msg:\"FILESTOREpdf\"; flow:established,to_server; fileext:\"pdf\"; filestore; sid:8; rev:1;)"

[0102] }

[0103] Wherein, raw_data_id represents the original data identifier, device_id and device_type represent the device to which the rule belongs, rule_file_type represents the rule format, file_change_status represents the change status of the rule file, and raw_content represents the raw rule content obtained from the collection.

[0104] After the data collection is completed, the platform associates and saves the device identifier, device type, rule format, collection time, change status, and original rule content with the collected device native rules to form a device native rule set. The device native rule set is then input into step S102, where step S102 performs field parsing and constructs a unified strategy object according to the corresponding rule parsing template.

[0105] Step S102: Parse the native rules of heterogeneous devices and construct a unified policy object. The platform takes the set of native rules formed in step S101 as input and selects the corresponding rule parsing template based on the device type, manufacturer type, and rule format associated with each device's native rules. The rule parsing template includes rule segmentation rules, field location rules, field type conversion rules, and field value mapping rules, used to convert the native rule syntax used by different security devices into a unified field representation.

[0106] For rule files containing multiple rules, the platform first segments them into independent native device rules based on line breaks, rule end characters, or device-specific delimiters. Then, using the corresponding rule parsing template, it extracts information such as device identifier, rule identifier, source address, source port, destination address, destination port, communication protocol, traffic direction, attack type, matching conditions, native handling actions, and activation status, and converts this information into a unified policy object. Fields not included in the native device rules are marked as null values; fields that can be determined by device access configuration records are supplemented with corresponding configuration values; and vendor-specific parameters that cannot be converted into unified fields are stored in extended fields to preserve the original rule information.

[0107] The platform standardizes field names and values ​​according to a pre-defined field mapping table. For example, native actions such as deny, drop, reject, and block are uniformly mapped to "block," allow and pass are uniformly mapped to "allow," and actions such as alert, log, and audit are mapped to "alarm" or "audit" based on device type. Simultaneously, arbitrary addresses, arbitrary ports, and protocol names in different rules are converted into a unified representation. Taking the Suricata rule as an example, alert, http, any, and fileext:"pdf" can be converted to matching conditions of "alarm," HTTP protocol, "any" address or port, and "file extension is PDF," respectively.

[0108] For example, taking the Suricata native rule in step S101 above as an example, the platform extracts the corresponding fields according to the Suricata rule parsing template and forms the following unified policy object:

[0109] {

[0110] "policy_id": "POLICY-IDS-0008",

[0111] "raw_data_id": "RAW-RULE-0001",

[0112] "device_id": "IDS-001",

[0113] "device_type": "Intrusion Detection System",

[0114] "rule_engine_type": "Suricata",

[0115] "rule_id": "8",

[0116] "rule_type": "File traffic detection rule",

[0117] "protocol": "HTTP",

[0118] "source_address": "any",

[0119] "source_port": "any",

[0120] "destination_address": "any",

[0121] "destination_port": "any",

[0122] "traffic_direction": "traffic direction to server",

[0123] "action": "alarm",

[0124] "match_conditions": {

[0125] "connection_state": "Connection established",

[0126] "file_extension": "pdf",

[0127] "file_store": true

[0128] },

[0129] "description": "Detect and save PDF files",

[0130] "rule_version": "1",

[0131] "enabled": true

[0132] }

[0133] Wherein, policy_id represents the unique identifier of the unified policy object, raw_data_id is used to associate with the device's native rule record in step S101, rule_engine_type represents the rule engine or rule format used by the native rule, rule_id represents the native rule identifier, protocol, address, port and traffic direction are used to describe the applicable traffic range of the rule, action represents the standardized handling action, match_conditions is used to store the specific detection conditions, rule_version represents the rule version, and enabled indicates whether the rule is currently enabled.

[0134] After the above analysis and standardization process, each device's native rule forms a corresponding unified policy object. The unified policy object includes at least the device identifier, rule identifier, device type, rule format, traffic matching range, matching conditions, standardized handling action, activation status, and extended fields. The platform aggregates the unified policy objects corresponding to each security device into a unified policy object set and inputs it into step S103 to determine the coverage of existing policies against the current attack and to construct the attack-defense status.

[0135] Step S103: Access Alarms and Construct Current Attack-Defense Status. The platform receives alarm data generated by heterogeneous security devices or security operation platforms and extracts information such as alarm identifier, source device, attack source, attack target, communication protocol, attack type, alarm level, alarm time, and attack characteristics. For multiple alarms with the same attack source, attack target, and attack type or similar attack characteristics within a preset time window, the platform aggregates them into a single alarm event and records the number of alarm occurrences and the highest alarm level to reduce the impact of duplicate alarms on subsequent strategy selection.

[0136] Within a preset time window, if multiple alarms have the same attack source, attack target, and attack type, they are considered to have the same attack conditions; if the attack target and attack type are the same, and the alarm rule identifier, key characteristics of the attack payload, or request path are the same, they are considered to have similar attack characteristics, and the alarms are aggregated into the same alarm event.

[0137] The platform uses aggregated alert events as an index to associate threat intelligence, network asset information, and device defense context. The device defense context includes security device protection relationships, device handling capabilities, device operating status, and the unified policy object set formed in step S102. Threat intelligence is used to determine the maliciousness level and intelligence credibility of the attack source; network asset information is used to determine the business attributes and importance level of the target asset; and the device defense context is used to determine the security devices capable of participating in the current attack handling and their existing defense capabilities. The platform queries threat intelligence based on the attack source address in the alert event to obtain the maliciousness level and intelligence credibility of the attack source; queries network asset information based on the attack target address to obtain the business type and importance level of the target asset; and filters security devices capable of participating in the current attack handling based on the protection relationship between security devices and the target asset, device operating status, and device handling capabilities.

[0138] It is worth explaining that "security device protection relationship" refers to the correspondence between a security device and the network area, business system, server, address range, or communication path it protects. For example: (1) WAF-001 protects Web server 10.10.1.20; (2) Firewall FW-001 controls communication from the external network to the DMZ area; (3) IDS-001 monitors traffic on the mirror port of the core switch; (4) Database auditing system DBA-001 audits database server 10.10.2.30; (5) IPS-001 is deployed in series on the communication path from the Internet egress to the server area. "Device handling capability" refers to which security operations the device itself can support. It is the capability boundary of the device, not the rules that have been configured. For example, (1) Firewalls support source IP, destination IP, port and protocol matching, and support allowing, blocking and rate limiting; (2) IDS supports traffic monitoring and alarms, but usually does not support active blocking; (3) WAF supports URL, request parameters and request header matching, and supports alarms and blocking; (4) Database auditing systems support database operation auditing, but may not support network traffic blocking.

[0139] The platform determines the policy coverage status of each security device against the current attack based on the matching relationship between the unified policy object and the current alarm event. The policy coverage status includes no policy coverage, detection coverage, alarm coverage, audit coverage, blocking coverage, and ineffective coverage. No policy coverage indicates that no policy is related to the current attack; detection, alarm, and audit coverage indicate that existing policies can identify or record the current attack but cannot actively block it; blocking coverage indicates that existing policies can prevent the current attack; and ineffective coverage indicates that a relevant policy exists, but it cannot form an effective defense due to the policy not being enabled or the device being in an abnormal state.

[0140] Among them, "existing policies" refers to the device native rules that have been configured on each target security device before the current alarm event occurred and have been parsed into a unified policy object in step S102. For example: (1) an existing firewall rule has already blocked a certain attack source, and the new action adds the same blocking rule, which is a duplication; (2) an existing rule allows a certain address to access, and the new action requires blocking the same address, which may constitute a conflict; (3) an existing rule blocks the entire network segment, and the new action only blocks one IP within the network segment, which may have an inclusion relationship.

[0141] The platform combines the above information into the current attack-defense status. , represented as:

[0142] ,

[0143] in, It indicates the current attack risk status, including attack type, alert level, threat intelligence credibility, and target asset importance level; It indicates the current defense capability status, including the device type, operating status, supported response actions, and the coverage of existing strategies against the current attack by the relevant security equipment.

[0144] The platform uses a preset encoding method to convert categorical data such as attack type, alarm level, device type, device operating status, supported action types, and policy coverage status into numerical representations. It also normalizes numerical data such as threat intelligence credibility and target asset importance level to form a structured attack-defense state that can be read by the reinforcement learning policy selection model. This attack-defense state serves as input for step S104, generating candidate device-policy actions and dynamic security action masks.

[0145] Step S104: Generate candidate device-policy action and dynamic security action mask. The platform uses the attack-defense state formed in step S103. As input, a set of candidate devices—policy actions—is generated from a preset policy action template library. The policy action template library is categorized according to security device type and is used to record the policy operation types, matching fields, handling actions, scope of application, and activation methods supported by different types of security devices.

[0146] Each candidate device-policy action consists of the target security device and its corresponding policy operation, represented as:

[0147] ,

[0148] in, Indicates the target security equipment; This indicates the operation type, such as adding a new strategy, modifying a strategy, or enabling an existing strategy. Indicates the policy matching conditions; This indicates actions such as alarms, audits, rate limiting, blocking, or denying access. Indicates the scope of the strategy; Indicates the duration for which the strategy is in effect.

[0149] The current set of candidate device-policy actions consists of multiple candidate device-policy actions:

[0150] ,

[0151] For each candidate device-policy action in the candidate device-policy action set, the platform generates a dynamic security action mask based on device adaptation constraints, policy consistency constraints, and business security constraints. Specifically, device adaptation constraints determine whether the target device supports the corresponding action, whether it protects the target asset, whether it is within a valid processing path, and whether it is currently available. Policy consistency constraints determine whether the candidate action duplicates or conflicts with existing policies in the unified policy object set. Business security constraints determine whether the scope of the candidate action and its expected business impact exceed a preset risk threshold. Note that expected business impact refers to the predicted normal business objects, normal business traffic range, and degree of impact that the action may affect before its actual issuance.

[0152] For candidate actions Its mask value Represented as:

[0153] ,

[0154] in, This indicates the result of the device adaptation constraint judgment. This indicates the result of the judgment regarding the policy consistency constraint. This indicates the result of the business security constraint assessment. All of the above constraints must be met. This indicates that the candidate action can be selected; if any constraint is not met, This indicates that the candidate action cannot be selected in the current state.

[0155] The platform generates a set of executable candidate devices and policy actions based on the mask results:

[0156] ,

[0157] For candidate actions with a mask value of 0, the platform masks them from the current candidate action space and records the corresponding constraint type and the reason for masking. During the sequential action selection process in step S105, after each candidate action is selected, the platform re-evaluates the three types of constraints based on the virtually updated defense state and dynamically updates the mask values ​​of the remaining candidate actions. Therefore, the dynamic security action mask changes with the current defense state and the selected actions, rather than being a pre-fixed static action list.

[0158] This step forms a set of candidate actions. Dynamic security action mask and set of executable candidate actions The information is then input into step S105, where the reinforcement learning policy selection model selects the cooperative device and the corresponding policy action.

[0159] Step S105: Selecting cooperative devices and corresponding policy actions based on reinforcement learning. The reinforcement learning sequential cooperative policy selection and structured cooperative candidate policy set generation process in step S105 is as follows: Figure 3 As shown. The platform will define the attack-defense status formed in step S103. and the set of executable candidate actions formed in step S104 The input reinforcement learning strategy selection model selects the security device and its corresponding strategy action to participate in the current attack handling from the candidate actions that satisfy the dynamic security action mask constraint.

[0160] In one implementation, the reinforcement learning policy selection model employs a state-action value function-based decision-making approach to calculate the action value of each executable candidate action in the current attack-defense state:

[0161] ,

[0162] in, The parameter is The state-action value function is used to evaluate candidate actions. In the current state The expected outcome of the action is determined by its value. A higher action value indicates a better overall expected outcome in terms of attack prevention, asset protection, and business impact.

[0163] In one specific implementation, the reinforcement learning policy selection model is implemented using a deep Q-network. The platform numerically encodes the current attack-defense state and each executable candidate device-policy action, respectively. The attack-defense state encoding and the candidate device-policy action encoding are input into the deep Q-network, and the state-action value corresponding to the candidate device-policy action is calculated through the input layer, hidden layer, and output layer. For candidate device-policy actions with a dynamic security action mask value of 0, they are not included in the action value comparison; for candidate device-policy actions with a dynamic security action mask value of 1, the one with the highest action value is selected as the current round device-policy action.

[0164] The platform selects the candidate action with the highest action value from the set of executable candidate actions:

[0165] ,

[0166] in, This represents the device-policy action selected in the current round. Since step S104 has already masked candidate actions with a mask value of 0 from the available action space, the reinforcement learning policy selection model only selects from actions that satisfy device adaptation constraints, policy consistency constraints, and business security constraints.

[0167] The above-mentioned The process of round state coding, dynamic safety action masking, state-action value calculation, and maximum action value selection is as follows: Figure 3 As shown in part (a).

[0168] To form a collaborative strategy executed by multiple heterogeneous security devices, the platform adopts a sequential action selection method. After selecting each device-policy action, the platform adds it to a structured collaborative candidate policy set and virtually updates the existing policy coverage status and device defense capability status based on the selected action to obtain the next round of attack-defense status.

[0169] ,

[0170] in, Indicates the first Attack / defense state before the round selection. Indicates the first Candidate actions selected in rounds, This represents the state transition function that updates the defense state based on the selected action. The virtual update is only used for subsequent action selection and does not directly modify the real security device policy in this step.

[0171] After completing the virtual update, the platform re-invokes the dynamic security action masking mechanism in step S104 to assess the feasibility of the remaining candidate actions and select the next device-policy action. In this way, the platform can dynamically adjust subsequent action selections based on the impact of the selected actions on the current defense capabilities, without needing to pre-enumerate all combinations of device and policy actions.

[0172] The set of executable candidate actions also includes an ending action. The platform terminates the sequence selection process when the reinforcement learning policy selection model selects an ending action, the updated policy coverage state meets the preset handling requirements, the number of selected actions reaches the preset upper limit, or there are no other executable candidate actions. If the existing policy can already meet the handling requirements of the current attack, the reinforcement learning policy selection model can directly select an ending action in the first round without generating a new policy change action.

[0173] After the sequence selection is completed, the platform combines the selected device-policy actions into a structured set of collaborative candidate policies according to the order of action selection:

[0174] ,

[0175] in, This represents the set of structured collaborative candidate strategies corresponding to the current alarm event. This indicates the number of strategy actions selected this time. The set records the target devices participating in the collaborative defense, strategy operation types, matching conditions, actions taken, scope of action, and duration of effect, and serves as the input for step S106 to perform the structured collaborative candidate strategy set verification. The process of forming a structured collaborative candidate strategy set by the selected devices and strategy actions in each round according to the selection order is as follows: Figure 3 As shown in part (b).

[0176] Step S106: Verify the structured collaboration candidate strategy set. The platform uses the structured collaboration candidate strategy set formed in step S105. As input, the device-policy actions are verified for completeness, executability, combination consistency, and business risk. The dynamic security action mask in step S104 is used to exclude individual candidate actions that are not executable before action selection. This step performs a final verification on the structured collaborative candidate policy set composed of multiple selected actions to avoid new conflicts or business risks arising after action combination.

[0177] The platform first performs an integrity check, checking whether each device-policy action contains necessary information such as target device, policy operation type, matching conditions, handling action, scope of effect and duration of effect, and whether the field types and value ranges conform to the unified policy object specification.

[0178] Secondly, perform an executability check to confirm that the target device is still available based on its current operating status and handling capabilities, and to support the corresponding matching fields, handling actions, and activation methods.

[0179] Subsequently, the platform performs a combined consistency check to determine whether there are duplicate, inclusion, overriding, or conflicting relationships between multiple device-policy actions and between the actions and existing policies. Actions that take opposite actions for the same traffic range, have contradictory policy scopes for different devices, or are inconsistent with existing high-priority policies are marked as conflicting. For actions with the same function and overlapping scopes, the action with the more accurate scope or the lower cost of change is retained.

[0180] The platform further verifies business risks. Based on the policy's scope of action, the target asset's importance level, affected ports, normal business traffic range, and management channel information, it determines whether the executable device-policy action may cause critical business interruption, widespread false blocking, or management channel unavailability. If the expected business risk exceeds the preset risk threshold, the business risk verification is deemed unsuccessful.

[0181] The verification result of the structured collaborative candidate strategy set can be expressed as:

[0182] ,

[0183] in, Indicates the integrity verification result. This indicates the executability check result. This indicates the result of the combinational consistency check. This indicates the results of the business risk verification.

[0184] It is worth explaining that when all the verification results are true, This indicates that the set of structured collaborative candidate strategies has passed the verification; if any verification result is false, The platform records the actions that fail verification and the reasons therefor, and deletes the corresponding actions, returns to step S105 to select again, or transfers the case to manual review.

[0185] Step S107: Convert the target device's native rules and distribute them collaboratively. The platform uses the structured collaborative candidate strategy set that passed the verification in step S106. As input, based on the target device identifier in each device-policy action, query the device access configuration record established in step S101 to obtain the corresponding target device rule template and policy distribution interface information.

[0186] The target device rule template describes the rule syntax, field names, field order, action keywords, value range, and necessary parameters supported by the target security device. The platform inputs unified fields such as the matching conditions, actions, scope, and duration of each device-policy action from the structured collaborative candidate policy set, along with the target device rule template, into the rule conversion module. The rule conversion module then generates native target device rules that the target security device can recognize. For the first... The rule transformation process for a single device-policy action is represented as follows:

[0187] ,

[0188] in, This indicates a device policy action that has passed verification. Indicates the target device The corresponding rule template, Represents a rule transformation function. This represents the target device's native rules obtained through the conversion.

[0189] In one implementation, the rule transformation function is implemented by a large language model. The platform inputs the device-policy action, the target device rule template, and the output format constraints into the large language model, which then performs the transformation from the unified fields to the device's native rule syntax. The large language model is only responsible for generating the native rule format and does not participate in target device selection or policy action decision-making.

[0190] After the native rules for the target device are generated, the platform performs syntax validation based on the target device rule template, checking whether the rule keywords, number of fields, field types, parameter formats, and necessary fields meet the requirements of the target device. For native rules for the target device that fail syntax validation, the platform records the erroneous fields and re-executes the rule conversion; if the validation still fails after reaching the preset number of retries, it is transferred to manual review and will not be automatically issued.

[0191] For native rules of target devices that pass syntax validation, the platform executes new policies, modifies existing policies, or enables existing policies based on the policy operation type, and sends them to the corresponding security devices through the policy distribution interface of the target device. When multiple security devices are involved, the platform distributes policies according to the action order or device dependency relationship in the structured collaborative candidate policy set; independent device-policy actions can be executed in parallel.

[0192] The platform records the target device, policy operation type, device response result, and distribution status of the native rules for each target device. If the interface call fails, the device refuses to execute, or the response times out, subsequent distribution operations that depend on the failed action are stopped, and the reason for the failure is recorded. This step outputs the native rules corresponding to each target device and their distribution results for step S108 to monitor the execution effect and provide feedback updates.

[0193] Step S108: Monitor execution results and update the reinforcement learning strategy selection model. Based on the strategy distribution results obtained in step S107, within a preset observation period after strategy distribution, the platform monitors whether the rules are successfully loaded and enabled, the number of strategy hits, changes in the number of attack alarms, the target device's operating status, the failure rate of normal business requests, and the business response time. It also confirms through the device query interface whether the distributed device native rules have been loaded and are enabled. If the strategy takes effect normally, attack alarms decrease, and normal business is not abnormal, the strategy is considered to have achieved the expected effect. If the attack continues, the device malfunctions, or normal business is significantly affected, a strategy rollback is triggered. If strategy distribution fails, the device response times out, or the rule does not actually take effect, the platform records the corresponding device, the device-strategy action, and the reason for the failure.

[0194] The platform compares the number of alarms, attack traffic, and policy hit rate before and after policy execution to determine whether the structured collaborative candidate policy set has achieved the expected handling effect. Simultaneously, it assesses the impact of the policy on normal business operations based on the normal request failure rate, business response time, critical service availability, and management channel connectivity. If attack alarms continue to occur, the policy fails to achieve the expected hit rate, or if normal business operations become abnormal, the scope of false blocking exceeds a preset threshold, device malfunctions, or multiple device policy conflicts occur, the execution is marked as abnormal. Based on the device's original rule version saved before issuance, the newly added rule is revoked, the modified rule is restored, or the original rule is reactivated.

[0195] The platform calculates the feedback reward for this collaborative strategy based on the effectiveness of attack handling, the degree of improvement in strategy coverage, business impact, cost of strategy change, and execution anomalies. It is represented as:

[0196] ,

[0197] in, Indicates the effect of attack handling; Indicates the degree of improvement in strategy coverage; This indicates the impact on normal business operations; Indicates the cost of strategy change; This indicates an abnormal execution result such as policy issuance failure, rule not taking effect, or triggering a rollback. , and to The preset weighting coefficients apply. A positive reward is given when an attack is successfully handled and does not affect normal business operations; a negative reward is given when the attack continues, business operations become abnormal, or a rollback is triggered.

[0198] The platform saves the attack-defense state before strategy selection, the structured collaborative candidate strategy set formed in step S105, the feedback reward, and the defense state after strategy execution as a strategy selection experience:

[0199] ,

[0200] in, This indicates the attack / defense state before strategy selection. This represents the set of selected collaborative strategies. This indicates a reward for performance feedback. This indicates the defensive state after the strategy is executed.

[0201] Since step S105 employs sequential action selection, the platform further breaks down the current structured collaborative candidate strategy set into selected device-strategy actions for each round according to the action selection order. It then forms action-level strategy selection experience by combining the virtual attack-defense state before each round's selection, the selected device-strategy action, the feedback reward, and the virtual attack-defense state after selection. The feedback reward is distributed to the selected device-strategy actions in each round according to a preset discount method, used to update the state-action value function in the reinforcement learning strategy selection model.

[0202] In this embodiment, the platform calculates feedback rewards based on the effectiveness of attack handling, the degree of policy coverage improvement, the impact on normal business, the cost of policy changes, and execution anomalies. Specifically, rewards are increased when an attack is successfully handled and policy coverage is improved; rewards are deducted when normal business is affected, a large number of rules or devices are changed, policy delivery fails, or a rollback is triggered. All rewards and penalties are combined according to preset weights.

[0203] The experience gained in strategy selection is written into a historical experience database. When the number of experiences reaches a preset update threshold, the platform uses the new experience to train the reinforcement learning strategy selection model offline or periodically update it. The updated model replaces the current model after it has been validated. The model training and exploration process does not directly affect production security equipment to avoid unverified strategies impacting normal operations. This forms a collaborative defense closed loop of "state construction—strategy selection—strategy issuance—execution monitoring—feedback update".

[0204] During the model update phase, the platform extracts action-level policy selection experience from the historical experience database, consisting of the pre-selection attack-defense state, selected device-policy action, feedback reward, and post-selection attack-defense state. It calculates the target action value based on the feedback reward and the state-action value of each executable candidate action in the next attack-defense state, and updates the model parameters of the reinforcement learning policy selection model based on the difference between the target action value and the model's output action value. The initial policy selection experience required for model training is generated from historical alarms in the isolated network topology, preset attack scenarios, and corresponding security device policy execution results; the model training and exploration process does not directly affect production security devices.

[0205] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0206] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning, characterized in that, Includes the following steps: Collect the native rules of heterogeneous security devices, and transform the native rules of the devices into a unified policy object according to a preset rule parsing template. Each security device in the heterogeneous security devices has a unique device identifier. When an alarm event is received, the alarm event is associated with threat intelligence, network asset information and device defense context to construct the attack-defense state corresponding to the current alarm event. The attack-defense state is used as input to generate candidate device-policy actions from the preset policy action template library. The corresponding dynamic security action mask is generated based on the candidate device-policy actions, and the set of executable candidate device-policy actions is obtained based on the dynamic security action mask. Input the attack-defense status and the set of executable candidate devices-policy actions, use the reinforcement learning policy selection model to select the security devices and corresponding policy actions to participate in the linkage, obtain the executable devices-policy actions and form a structured collaborative candidate policy set; The structured collaborative candidate strategy set is validated, and the validated structured collaborative candidate strategy set is converted into device native rules that can be recognized by the target security device. The device native rules are then sent to one or more corresponding target security devices, and the strategy sending results are recorded. The target security device loads and executes the issued native device rules, monitors the execution results of the target security device after executing the native device rules, calculates feedback rewards based on the execution results and generates policy selection experience, and uses the policy selection experience to update the reinforcement learning policy selection model.

2. The method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The process involves collecting the native rules of heterogeneous security devices and converting them into a unified policy object according to a preset rule parsing template. Specifically: Based on the access configuration records of heterogeneous security devices, the native rules currently configured for each security device are obtained through the device management interface, rule file reading interface, or other preset collection methods. The access configuration records include device identifier, device type, manufacturer type, management address, rule format, rule acquisition method, rule parsing template, target device rule template, and policy distribution interface information. For security devices with different deployment methods, the data collection component is invoked to read the device's native rules. At the same time, the changed native rules of the device are identified by rule version, update time, or file hash value. The access configuration records, collection time, and rule file change status are associated and saved with the device's native rules to obtain the device's native rule set; The system parses the set of native rules for each device. Based on the device type, manufacturer type, and rule format associated with each device's native rules, it selects the appropriate rule parsing template, parses the fields of the set of native rules for each device according to the rule parsing template, and constructs a unified policy object. The rule parsing template includes rule segmentation rules, field location rules, field type conversion rules, and field value mapping rules.

3. The method for selecting and distributing cooperative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, When an alarm event is received, the alarm event is associated with threat intelligence, network asset information, and device defense context to construct the attack-defense state corresponding to the current alarm event, specifically as follows: The system receives alarm data generated by heterogeneous security devices or security operation platforms, extracts alarm information from the alarm data, and forms alarm events, including alarm identifier, source device, attack source, attack target, communication protocol, attack type, alarm level, alarm time, and attack characteristics. For multiple alarm messages with the same attack conditions or similar attack characteristics within a preset time window, the alarm information is aggregated into the same alarm event, and the number of alarm occurrences and the highest alarm level are recorded. The attack conditions include attack source, attack target, and attack type. The alarm events are used as an index to associate threat intelligence, network asset information, and device defense context. The device defense context includes security device protection relationships, device handling capabilities, device operating status, and unified policy objects. The threat intelligence is used to determine the malice level and intelligence credibility of the attack source. The network asset information is used to determine the business attributes and importance level of the target asset. The device defense context is used to determine the security devices that can participate in the current attack handling and their existing device defense capabilities. Based on the matching relationship between the unified policy object and the current alarm event, the policy coverage status of each security device for the current attack is determined. The policy coverage status includes no policy coverage, detection coverage, alarm coverage, audit coverage, blocking coverage, and failure coverage. By using alarm events, threat intelligence, and network asset information as the current attack risk status, and the unified policy object, device defense context, and policy coverage status as the current defense capability status, the attack-defense status at the current moment is constructed.

4. The method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The preset policy action template library is classified according to the type of security device and is used to record the policy operation type, matching field, handling action, scope of application and activation method supported by different types of security devices; Each candidate device-policy action consists of the target security device and its corresponding policy operation, as shown in the following formula: , in, Indicates the target security equipment. Indicates the operation type, including adding a policy, modifying a policy, or enabling an existing policy. Indicates the policy matching conditions. This indicates the actions taken, including alerting, auditing, limiting speed, blocking, or denying access. Indicates the scope of the strategy. Indicates the duration for which the policy is in effect; Multiple candidate device-policy actions constitute the candidate device-policy action set at the current moment.

5. The method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The process of generating a corresponding dynamic security action mask based on the candidate device-policy action is as follows: For each candidate device-policy action, a security action mask is generated based on device adaptation constraints, policy consistency constraints, and business security constraints. For each candidate device-policy action, its security action mask is... Represented as: , in, This indicates the result of the device adaptation constraint judgment. This indicates the result of the judgment regarding the policy consistency constraint. This indicates the result of the judgment on business security constraints. This represents the logical AND operation; The set of executable candidate device-policy actions is obtained based on the mask value. Candidate device-policy actions with a mask value of 0 are masked in the current candidate action space, and the corresponding constraint type and masking reason are recorded. During the sequential action selection process, after each candidate device-policy action is selected, the above three types of constraints are re-evaluated based on the virtual updated attack-defense status, and the mask values ​​of the remaining candidate device-policy actions are dynamically updated.

6. The method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning strategy selection model adopts a decision-making method based on the state-action value function, calculates the action value of each executable candidate device-strategy action in the current attack-defense state, and selects the executable candidate device-strategy action with the highest action value from the set of executable candidate device-strategy actions, that is, obtains the executable device-strategy action; The input attack-defense status and executable candidate device-policy action set are used to select participating security devices and corresponding policy actions using a reinforcement learning policy selection model, thereby obtaining executable device-policy actions and forming a structured collaborative candidate policy set. Specifically: Using a reinforcement learning policy selection model, the executable candidate device-policy action with the highest action value is selected from the executable candidate device-policy action set to obtain the executable device-policy action. The executable device-policy action is added to the set to obtain a structured collaborative candidate policy set. Based on the selected executable device-policy action, the existing policy coverage state and device defense capabilities are virtually updated to obtain the attack-defense state for the next round.

7. The method for selecting and distributing cooperative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The structured collaborative candidate strategy set also includes an ending action. The selection process is terminated if the reinforcement learning strategy selection model selects an ending action, the updated strategy coverage state meets the preset handling requirements, the number of selected actions reaches the preset upper limit, or there are no other executable candidate actions. If the existing strategy is sufficient to handle the current attack, the reinforcement learning strategy selection model will directly select the termination action in the first round and stop generating new strategy change actions.

8. The method for selecting and distributing collaborative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The process of validating the structured collaborative candidate strategy set involves converting the validated set into device-native rules that can be recognized by one or more target security devices, and then distributing these device-native rules to the corresponding target security devices. Specifically: Perform the following risk checks on the structured collaborative candidate strategy set in sequence, and obtain the check results, including: Integrity verification: Check whether each executable device-policy action in the structured collaborative candidate policy set has the necessary information, and whether the field types and value ranges conform to the unified policy object specification; the necessary information includes target device, policy operation type, matching conditions, disposal action, scope of effect and duration of effect; Executability verification: Based on the current operating status and handling capabilities of the target device, confirm that the target device is still available and supports the corresponding matching fields, handling actions, and activation methods; Combinatorial consistency verification: Determine whether there are duplicate, inclusion, overriding or conflicting relationships between multiple executable device-policy actions, or between the actions and existing policies, and obtain the verification result; the existing policies are the device native rules that were configured on each target security device before the current alarm event occurred and were parsed into a unified policy object. Business risk verification: Based on the policy's scope of action, the target asset's importance level, affected ports, normal business traffic range, and management channel information, determine whether the executable device-policy action may cause critical business interruption, large-scale false blocking, or management channel unavailability; if the expected business risk exceeds the preset risk threshold, the expected business is marked as a high-risk policy. The verification result is represented as: , in, Indicates the integrity verification result. This indicates the executability check result. This indicates the result of the combinational consistency check. This indicates the results of the business risk verification. When the verification result is true, the structured collaborative candidate strategy set passes the verification; otherwise, the action that fails the verification and its reason are recorded, the corresponding executable device-strategy action is deleted, and the reinforcement learning strategy selection model is used to continue the selection or the case is transferred to manual review. The structured collaborative candidate policy set that has passed the verification is used as input. The access configuration record of the device is queried according to the device identifier to obtain the corresponding target device rule template and policy distribution interface information. The target device rule template is used to describe the rule syntax, field name, field order, action keyword, value range and necessary parameters supported by the target security device. Based on the unified fields in the executable device-policy action and the target device rule template, generate native device rules that the target security device can recognize. For the first... An executable device-policy action, whose rule transformation process is represented as follows: , in, This indicates an executable device-policy action that has passed verification. Indicates target security equipment The corresponding target device rule template, Represents a rule transformation function. This represents the native device rules of the target device obtained after the conversion; The policy is sent to the corresponding security device through the policy distribution interface of the target device. When multiple security devices are involved, the policy is distributed according to the action order or device dependency relationship in the structured collaborative candidate policy set. The independent executable devices and policy actions are executed in parallel.

9. The method for selecting and distributing cooperative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 8, characterized in that, The rule conversion function adopts a large language model, which will complete the conversion from unified fields to the device's native rule syntax; Once the native rules for the target security device are generated, syntax validation is performed based on the target device rule template to check whether the keywords, number of fields, field types, parameter formats, and necessary fields in the rules meet the requirements of the target device. For native device rules that fail syntax validation, the error fields are recorded and the rule conversion is re-executed. If the validation still fails after the preset number of retries, the rule is transferred to manual review and automatic distribution is stopped. For native device rules that pass syntax validation, execute new policies, modify policies, or enable existing policies according to the policy operation type, and send them to the corresponding security devices through the policy distribution interface of the target device. Record the target device, policy operation type, device response result, and distribution status corresponding to the native rules of each device; if the interface call fails, the device refuses to execute, or the response times out, stop subsequent distribution operations that are dependent on the failed action, and record the reason for the failure.

10. The method for selecting and distributing cooperative defense strategies for heterogeneous security devices based on reinforcement learning according to claim 1, characterized in that, The target security device loads and executes the issued native device rules, monitors the execution results of the target security device after executing the native device rules, calculates feedback rewards based on the execution results and generates policy selection experience, and uses the policy selection experience to update the reinforcement learning policy selection model, specifically: If the distribution is successful and the target device is operating normally, perform the following steps: Within a preset observation period, monitor the policy hit rate, attack blocking status, device operation status, and normal business operation status of the target security device, and confirm whether the issued device native rules have been loaded and are enabled through the device query interface. Compare the number of alarms, attack traffic, and policy hit rate before and after policy execution to determine whether the structured collaborative candidate policy set has achieved the expected handling effect. At the same time, judge the impact of the policy on normal business based on the normal request failure rate, business response time, critical service availability, and management channel connectivity status. If attack alerts continue to occur, the policy does not produce the expected hit, normal business is abnormal, the scope of false blocking exceeds the preset threshold, the device is operating abnormally, or there is a policy conflict between multiple devices, then this execution will be marked as abnormal, and the newly added rule will be revoked, the modified rule will be restored, or the original rule will be reactivated according to the original rule version of the device saved before the issuance. The feedback reward for this collaborative strategy is calculated based on the attack handling effect, the degree of policy coverage improvement, the business impact, the cost of policy change, and the execution anomaly. A positive reward is obtained when the attack is successfully handled and does not affect normal business, and a negative reward is obtained when the attack continues, business anomalies occur, or a rollback is triggered. Save the attack-defense state before strategy selection, the set of structured collaborative candidate strategies, the feedback reward, and the defense state after strategy execution as a strategy selection experience. According to the action selection order, the current structured collaborative candidate strategy set is divided into the selected device-strategy action in each round. The virtual attack-defense state before each round of selection, the selected device-strategy action, the feedback reward, and the virtual attack-defense state after selection are formed into action-level strategy selection experience. The feedback reward is distributed to the selected executable device-strategy action in each round according to a preset discount method, which is used to update the state-action value function in the reinforcement learning strategy selection model. The action-level policy selection experience is written into the historical experience database. When the number of experiences reaches the preset update condition, the newly added experience is used to train the reinforcement learning policy selection model offline or update it periodically. The updated model replaces the current model after it has been verified. For cases where the distribution fails, the device response times out, or the rule does not actually take effect, the execution is marked as abnormal and a negative reward is given to form corresponding strategy selection experience. At the same time, the corresponding target security device, executable device-policy action, and failure reason are recorded.