A Multi-Dimensional Security Protection Effectiveness Verification Method Based on the ATT&CK Framework
By constructing a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework, the standardization problem of security assessment methods in existing technologies is solved. This enables multi-dimensional quantitative assessment of security devices and identification of protection blind spots, provides priority-based remediation suggestions, and ensures the rational allocation of security hardening resources.
Patent Information
- Application Number
- CN202611351107.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing security assessment methods lack a systematic binding with the ATT&CK framework's standardized classification system, making it difficult to compare the protection results of various security devices horizontally according to a unified standard, failing to identify protection weaknesses in the attack chain, and failing to allocate security hardening resources reasonably.
A multi-dimensional security protection effectiveness verification method based on the ATT&CK framework is constructed. Attack cases are labeled with attack knowledge graphs, simulated attack behaviors are executed, verification results are collected, and aggregated analysis is performed based on ATT&CK attack stage, security device type, and network area to generate a protection assessment report.
It enables multi-dimensional quantitative assessment of security protection capabilities, identifies protection blind spots, provides priority-based remediation suggestions, and ensures the rational allocation of security hardening resources.
Smart Images

Figure CN123122936A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of network security technology, and specifically to a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework. Background Technology
[0002] Enterprise network security protection systems typically consist of various security devices such as firewalls, IDS / IPS, EDR, and WAF, covering different network areas such as the DMZ zone, core zone, production zone, and office zone.
[0003] Existing security assessment methods typically employ penetration testing or vulnerability scanning, lacking a systematic binding with the ATT&CK framework's standardized classification system. Protection results for various devices are recorded independently, making it difficult to perform cross-sectional aggregation analysis according to a unified standard.
[0004] The above approach makes it impossible to compare the actual interception and alarm detection capabilities of various security devices against different attack techniques, lacks a quantitative description of the overall protection level of different network areas, and fails to systematically identify the stages in the attack chain where there are protection weaknesses. This leads to the technical problem that security hardening resources cannot be allocated reasonably according to priority. Summary of the Invention
[0005] This invention provides a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework, which solves the technical problems in related technologies such as the lack of multi-dimensional quantitative evaluation of security protection capabilities and the inability to systematically identify protection blind spots from dimensions such as attack stage, device type and network area.
[0006] This invention discloses a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework, comprising the following steps: An attack knowledge graph is constructed, structured data from the ATT&CK framework is obtained, and attack cases are labeled according to four dimensions: attack stage, technology, sub-technology, and attack organization. The execution script, target platform, and precondition metadata of the attack cases are stored together with the multi-dimensional label information in the attack knowledge graph database. Execute attack simulations and collect execution results. Generate an attack task queue based on the attack verification plan. Drive the attack simulation nodes to perform simulated attack behaviors in the target network environment. Collect verification results, blocking status, alarm status, and associated security device type and network area for each attack case. Store these data in association with the case node in the attack knowledge graph. Aggregate the execution results data using the ATT&CK attack phase as the grouping key, and calculate the penetration risk index for each attack phase. ,in , To verify that the result represents the number of valid test cases, The total number of test cases; the blocking rate is calculated by cross-aggregating the execution result data using security device type and ATT&CK technology number as two-dimensional grouping keys. and detection rate ; The execution result data is aggregated using the network region identifier as the grouping key, and the basic score of each network region is calculated. Alarm coverage coefficient and final protection score By integrating the above multi-dimensional statistical analysis results, a security protection effectiveness assessment report is generated.
[0007] Furthermore, the penetration risk indicator The calculation also includes: based on The range of values is used to mark the risk level of each attack phase. When marked as high risk, The time is marked as medium risk, when Time-marked as low risk; among them, test cases with abnormal verification results are included in the total number of test cases, but not in the remaining test cases. The numerator or denominator.
[0008] Furthermore, the blocking rate and detection rate The calculation also includes: and Safety equipment types and technology combinations that are all below the preset threshold are marked as serious protection blind spots.
[0009] Furthermore, the area protection score Calculate as follows: ; ; ; in, This represents the number of test cases in this network area whose verification results are invalid. To verify that the result represents the number of valid test cases, To verify the number of test cases where the result was valid but triggered an alarm, This is the alarm weighting coefficient.
[0010] Furthermore, it also includes a scoring alarm triggering step: when the protection score of a certain network area shows a downward trend for three consecutive days, or when the daily score drop exceeds a preset threshold, an alarm notification is generated.
[0011] Furthermore, the remediation priority recommendations in the assessment report also include: calculating the weighted risk score of each protection weakness based on comprehensive penetration risk indicators, blocking rate gaps, network area ratings, and regional importance weights, and generating a list of remediation recommendations ranked by priority from high to low scores.
[0012] Furthermore, it also includes the step of mutating and expanding attack cases: performing structured parsing on the execution script of each attack case in the attack knowledge graph, extracting five categories of mutable elements: string constants, encoding methods, execution order, network protocol field values, and payload partitioning strategies, and generating a description structure of the use case mutation space. ;based on Five types of mutation operators—string encoding replacement, parameter obfuscation, execution order substitution, payload fragmentation, and protocol field randomization—are applied to generate mutant records. Submit the original test cases and variant test cases to the attack simulation node for execution, collect the execution results, and associate and store them with the original test case number, operator type, and ATT&CK technology number.
[0013] Furthermore, it also includes the step of calculating the robustness index of the rules: grouping the execution results of mutated test cases by the original test case number and calculating the overall mutation escape rate. Classification escape rate by mutation operator type ,in , This represents the number of valid validation results in the variant test cases. To verify that the result represents the number of failures, identify the mutation operator type with the highest classification escape rate in each original use case. ,Will With each Co-encapsulation generates rule robustness index vector .
[0014] Furthermore, it also includes the step of generating a multi-dimensional robustness distribution matrix: converting the regular robustness index vector... According to ATT&CK technical number and types of safety equipment Perform cross-aggregation to calculate the average mutation escape rate. When the blocking rate of a certain combination Not lower than the high threshold of blocking rate and Not lower than the high threshold of mutation escape rate When this condition is met, the combination is marked as a surface-safe-substantially-fragile state. ; Calculate the escape rate distribution of each mutation operator type in each combination, and output the most vulnerable mutation operator type. .
[0015] This invention discloses a multi-dimensional security protection effectiveness verification system based on the ATT&CK framework, comprising: The knowledge graph construction module is used to acquire structured data from the ATT&CK framework, label attack cases according to four dimensions: attack stage, technology, sub-technology, and attack organization, and generate and store the attack knowledge graph database. The attack simulation execution module is used to generate an attack task queue based on the attack verification plan, drive the attack simulation node to perform simulated attack behavior in the target network environment, collect the verification results, blocking status, alarm status and context information of each attack case, and associate and store them with the case node in the attack knowledge graph. The attack phase analysis module aggregates execution result data using the ATT&CK attack phase as the grouping key, and calculates the penetration risk index for each attack phase. and the percentage of each result type; The device protection analysis module is used to cross-aggregate the execution result data using security device type and ATT&CK technology number as two-dimensional grouping keys, and calculate the blocking rate of each device type against each attack technique. and detection rate Identify critical blind spots in protection; The regional scoring module is used to aggregate the execution result data with the network region identifier as the grouping key, calculate the basic score, alarm coverage coefficient and final protection score of each region, store the time-series scoring data and trigger scoring alarms; The report generation module integrates the outputs of the attack phase analysis module, device protection analysis module, and regional scoring module to generate a security protection effectiveness assessment report that includes ATT&CK attack phase protection effectiveness evaluation, security device type protection capability evaluation, network area comprehensive protection score and trend analysis, security protection risk ranking, and remediation priority recommendations.
[0016] The beneficial effects of this invention are as follows: This invention binds and stores the multi-dimensional tagging system of the ATT&CK framework with the results of attack simulation execution, and calculates penetration risk indicators based on three dimensions: attack stage, security device type, and network region. Blocking rate Detection rate and regional protection score It solves the technical problems that the effectiveness of protection is difficult to objectively quantify according to a unified standard and that security hardening resources cannot be reasonably allocated according to priority. It achieves the technical effect of being able to systematically identify the protection shortcomings of the attack chain, accurately locate the protection gaps of specific equipment types against specific attack technologies, and output priority-based repair suggestions according to business importance. Attached Figure Description
[0017] Figure 1 This is a flowchart of a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework provided in an embodiment of the present invention; Figure 2 This is a sub-flowchart of the multi-dimensional security protection effectiveness verification method based on the ATT&CK framework provided in the embodiments of the present invention, which executes attack simulation tasks and collects execution results. Detailed Implementation
[0018] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.
[0019] Example 1 At least one embodiment of the present invention discloses a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework, see [link to relevant documentation]. Figures 1-2 This includes the following steps: Step 1: Construct an ATT&CK attack knowledge graph.
[0020] The system acquires structured data from the ATT&CK framework and imports the three-tiered structure of Tactics, Techniques, and Sub-Techniques into the attack knowledge base management module, storing it as localized structured data in JSON format. Each technique entry includes fields such as technique ID, technique name, list of associated tactics, list of sub-technique IDs, associated threat organization information, and platform applicability.
[0021] For each attack case, a YAML configuration file is used to describe it, and the attack cases are tagged according to the following four dimensions: The first dimension is the attack phase, corresponding to the ATT&CK tactical layer. It is labeled with the English names and numbers of the tactics and covers 14 tactical phases, including initial access (TA0001), execution (TA0002), persistence (TA0003), privilege escalation (TA0004), defense and evasion (TA0005), credential access (TA0006), discovery (TA0007), lateral movement (TA0008), collection (TA0009), command and control (TA0011), data leakage (TA0010), and impact (TA0040). One attack use case can be associated with multiple attack phase tags.
[0022] The second dimension is technology, which uses ATT&CK technology numbers (in the form of TXXXX) and corresponding technology names to establish a many-to-many mapping relationship between attack cases and technologies.
[0023] The third dimension is sub-technology, which is labeled with ATT&CK sub-technology number (in the form of TXXXX.XXX) and corresponding sub-technology name. Sub-technology belongs to its parent technology and provides a more granular technical description. This field is left empty for technical entries that do not yet have sub-technology subdivisions.
[0024] The fourth dimension is the attacking organization, which uses the organization number (in the form of GXXXX) and organization name in ATT&CK to record the identification information of known threat organizations or APT organizations.
[0025] The aforementioned multi-dimensional tag information, along with metadata such as the execution scripts, target platforms, and preconditions of attack cases, are stored together in the attack knowledge graph database. The attack knowledge graph database contains the following node types: attack case nodes (CaseNode), tactical nodes (TacticNode), technique nodes (TechniqueNode), sub-technique nodes (SubTechniqueNode), and threat organization nodes (GroupNode). The edge relationship types between nodes include: use case belongs to (BELONGS_TO) tactic, use case implements (IMPLEMENTS) technique, technique is subdivided into (HAS_SUB) sub-technique, and threat organization uses (USES) technique.
[0026] It should be noted that the aforementioned attack knowledge graph database can be stored using a relational database or graph database that supports graph structure queries. Graph structure data supports fast querying and aggregation by any dimension, enabling subsequent multi-dimensional statistical analysis to access the required tag-related data through a unified graph query interface.
[0027] Step 2: Execute the attack simulation task and collect the execution results.
[0028] Based on the attack knowledge graph, an attack verification plan is configured, which includes the target network environment (IP addresses or network segments of attacker nodes and target nodes), a set of attack cases, an execution time window, and a limit on the number of concurrent executions. An attack task queue is generated and distributed to attack simulation nodes for execution.
[0029] The attack simulation node executes simulated attack behaviors in the target environment according to the execution script in the attack case configuration file, while monitoring the following three types of signals in real time to determine the execution result: First, network layer blocking signals, which determine whether the attack traffic has been blocked by network security devices by monitoring whether the TCP connection has been reset (RST message) and whether the HTTP response code is a security device characteristic response code; Second, host layer blocking signals, which determine whether the attack payload has been blocked by endpoint security software by monitoring host behavior results such as whether process creation was successful and whether file writing was blocked; Third, alarm signals, which determine whether the security device has generated corresponding alarm events by querying alarm logs in real time through API integration with the SIEM platform.
[0030] For each completed attack test case, the following execution result data fields are collected and stored: (1) Result: an enumeration type with three values: effective (the attack successfully bypassed all protection measures), ineffective (the attack was successfully blocked by the protection device), and anomaly (an unexpected error occurred during the execution process, and the result is unreliable).
[0031] (2) Block Status: Boolean type, records whether attack traffic or behavior is blocked by security devices.
[0032] (3) AlertStatus: Boolean type, records whether the attack simulation triggered an alarm event of the security device.
[0033] (4) Context information, including execution timestamp (ExecuteTime), attacker node identifier (AttackerNode), target node identifier (TargetNode), associated security device type (DeviceType, such as WAF, IDS / IPS, EDR, firewall, etc.) and network zone (such as DMZ zone, production zone, office zone, core zone, etc.).
[0034] The execution result data is associated with and stored in the attack knowledge graph through foreign key association, forming a complete dataset that can be used for multi-dimensional aggregation analysis.
[0035] It should be noted that, among the three states of the above verification results, the test cases in the abnormal state are only counted in the total number of test cases in the subsequent statistical analysis, and are not included in the numerator or denominator of the protection effectiveness rate, blocking rate, detection rate and regional protection score, so as to avoid the abnormal execution environment from interfering with the statistical results.
[0036] Step 3: Calculate penetration risk indicators based on the ATT&CK attack phase.
[0037] Using the ATT&CK attack phase as the grouping key, aggregate the execution results data and calculate the following statistical indicators for each attack phase.
[0038] The total number of use cases is calculated using the following formula: ; in, This represents the number of valid test cases verified during this attack phase. This represents the number of test cases whose verification results were invalid during this attack phase. This represents the number of test cases whose verification results are abnormal during this attack phase.
[0039] The success rate, failure rate, and anomaly rate are calculated using the following formulas: ; Penetration Risk Indicators Defined as the proportion of valid test cases during this attack phase out of the total number of test cases, calculated using the following formula: ; The physical meaning of is the probability that this attack phase can be successfully bypassed. Based on The range of values is used to label the risk level of each attack phase: when When marked as high risk; The time is marked as medium risk; when The time is marked as low risk.
[0040] In this embodiment of the application, in order to enable security operations and maintenance personnel to intuitively identify the weakest stage in the attack chain, the above statistical results are arranged in the tactical defense view screen according to the standard attack chain order of the ATT&CK framework, from the initial access to the impact, and each stage corresponds to a statistical card, displaying the total number of test cases, the number and proportion of each result type in a pie chart, and the penetration risk index value, and high, medium and low risk stages are visually marked with red, orange and green.
[0041] Step 4: Calculate the blocking rate and detection rate based on the type of security device.
[0042] Using the security device type (DeviceType) and ATT&CK technology ID (TechniqueID) as dual-dimensional grouping keys, the execution result data is cross-aggregated to calculate the two core indicators of blocking rate and detection rate.
[0043] Blocking rate Indicates the type of safety device ATT&CK attack techniques The interception effectiveness is calculated using the following formula: ; in, For devices with a blocking status of true and an associated device type of Attack techniques are The number of use cases; For the associated device type And the attack technique is The total number of executed test cases.
[0044] Detection rate Indicates the type of safety device ATT&CK attack techniques The alarm detection capability is calculated using the following formula: ; in, For alarm status is true and associated device type is Attack techniques are The number of use cases.
[0045] It should be noted that the blocking rate and the detection rate are independent of each other. A device may generate an alarm while blocking an attack, or it may detect an attack but not block it, or neither block it nor generate an alarm (this represents the most severe lack of protection). For and Equipment types and technology combinations that account for less than 20% are marked as serious protection blind spots.
[0046] In this embodiment of the application, the above cross-statistical results are displayed in a three-dimensional tower plot in the security protection view, with the X-axis representing the security device type, the Y-axis representing the ATT&CK attack technique category, and the Z-axis corresponding to... and The two values are distinguished by different colors, and the types of equipment and technology combinations in serious protection blind spots are highlighted in red. It also supports interactive operations such as rotating, scaling and filtering by attack stage of the 3D view.
[0047] Step 5: Calculate the regional protection score based on network regions.
[0048] Using the Network Zone identifier as the grouping key, the execution result data is grouped and aggregated at the zone level, and the protection score for each network zone is calculated according to the following method. The scoring range is from 0 to 100, with lower scores indicating greater safety risks.
[0049] First, calculate the base score. The basic protection effectiveness rate is determined by the percentage of failed use cases among use cases with clearly defined results. ; in, This represents the number of test cases in this network area whose verification results are invalid. This represents the number of valid test cases within this network region. The denominator only includes test cases with clear results and excludes abnormal test cases.
[0050] Secondly, calculate the alarm coverage factor. This is used to correct the baseline score, reflecting the value of the device's detection capability when it fails to block an attack: ; in, This represents the number of test cases in this network area that had valid verification results but triggered alarms. This is the alarm weighting factor, with a default value of 0.1, meaning that even if the alarm is not blocked but is detected in time, it can provide a maximum score correction of 10%.
[0051] The final area protection score calculation formula is: ; The final score is capped at 100 to prevent the corrected score from exceeding the range. Based on The range of values is used to rate the protection status of each area: when The rating is good; when The rating is medium; when The rating is a warning; when The rating is dangerous.
[0052] It should be noted that the above network zone division supports custom configuration according to the actual network plan of the enterprise. Typical zone divisions include DMZ zone, production zone, office zone, core zone, R&D zone, etc. The division of each zone is determined by the NetworkZone field in the attack simulation execution record to determine the zone to which the target node belongs.
[0053] In this embodiment, to track the changing trend of the protection capabilities of each network area over time, a daily scheduled task triggers the calculation of regional protection scores. The daily score results are stored in a time-series data store in key-value pairs of region identifier and date, and a line graph is used to display the changes in the protection scores of each network area over the past 7 days. Based on step 5, the following score alarm triggering step is also included: when the score of a certain area shows a downward trend for 3 consecutive days, or the score drops by more than 15 points in a single day, an alarm notification is generated to prompt security management personnel to pay attention to the changes in the security situation of that area.
[0054] Step 6: Generate an evaluation report by integrating the results of multi-dimensional analysis.
[0055] Integrating the multi-dimensional statistical analysis results from steps 3 to 5, a security protection effectiveness assessment report is generated. The report is output in both PDF and HTML formats. The core content of the report includes the following five parts: (1) ATT&CK attack phase protection effectiveness assessment: The penetration risk index values of 14 ATT&CK attack phases are summarized and presented in the form of radar charts. The attacks are sorted from highest to lowest risk of penetration, and the vulnerable attack phases and their specific effective, ineffective, and abnormal use cases are listed in detail.
[0056] (2) Security equipment type protection capability assessment: Summarize the average blocking rate and average detection rate of each security equipment type when dealing with all ATT&CK attack techniques, identify the equipment types with protection blind spots in specific attack technique categories, and output specific technical coverage gap analysis.
[0057] (3) Network Area Comprehensive Protection Score and Trend Analysis: The current protection score of each network area is listed in tabular form. The rating and the score trend over the past 7 days are highlighted, with areas under warning and danger ratings clearly marked, along with a line chart of the score over the past 7 days.
[0058] (4) Top 3 Security Protection Risk Ranking: Output a Top 3 high-risk list separately for security protection equipment type and ATT&CK attack stage, based on the penetration risk index value. and blocking rate gap Quantitative indicators are used as the basis for ranking.
[0059] (5) Repair priority recommendations: Based on the results of multi-dimensional analysis, the weighted risk score of each protection weakness is calculated by comprehensively considering the penetration risk index, the blocking rate gap, the network area rating and the weight of regional importance (core area weight 1.5, production area weight 1.3, DMZ area weight 1.1, office area weight 1.0). A repair recommendation list is generated in order of priority from high to low score. Each recommendation includes the repair target, the current risk level and the recommended repair measures.
[0060] It should be noted that the above report generation can be automatically triggered after the attack verification plan is completed, or it can be manually triggered by the user. Both triggering methods use the same multi-dimensional statistical aggregation logic and the output content is consistent.
[0061] On the other hand, this invention also proposes a multi-dimensional security protection effectiveness verification system based on the ATT&CK framework, including: The knowledge graph construction module is used to acquire structured data from the ATT&CK framework, label attack cases according to four dimensions: attack stage, technology, sub-technology, and attack organization, and generate and store the attack knowledge graph database. The attack simulation execution module is used to generate an attack task queue based on the attack verification plan, drive the attack simulation node to perform simulated attack behavior in the target network environment, collect the verification results, blocking status, alarm status and context information of each attack case, and associate and store them with the case node in the attack knowledge graph. The attack phase analysis module is used to aggregate the execution result data with the ATT&CK attack phase as the grouping key, calculate the total number of test cases, the proportion of each result type and the penetration risk index for each attack phase, and display them in the tactical defense view screen in the order of the attack chain. The device protection analysis module is used to cross-aggregate the execution result data using security device type and ATT&CK technology number as two-dimensional grouping keys, calculate the blocking rate and detection rate of each device type against each attack technology, identify serious protection blind spots, and visualize them in a three-dimensional tower diagram. The regional scoring module is used to aggregate the execution result data with the network region identifier as the grouping key, calculate the basic score, alarm coverage coefficient and final protection score of each region, store the time-series scoring data and trigger scoring alarms; The report generation module integrates the outputs of the attack phase analysis module, device protection analysis module, and regional scoring module. It generates a security protection effectiveness assessment report in both PDF and HTML formats, consisting of five parts: ATT&CK attack phase protection effectiveness assessment, security device type protection capability assessment, network area comprehensive protection score and trend analysis, top 3 security protection risk ranking, and remediation priority recommendations.
[0062] By way of example, the system proposed in this invention may also include various features and combinations thereof in the method embodiments, which will not be elaborated here.
[0063] This invention binds and stores the multi-dimensional tag system (attack phase, technology, sub-technology, attack organization) of the ATT&CK framework with the execution results of attack cases, so that the execution results of each attack simulation can be aggregated and analyzed according to any ATT&CK dimension, thus overcoming the problem of horizontal comparison difficulties caused by the lack of uniformity in attack behavior classification standards in traditional security assessments.
[0064] Penetration Risk Indicators The proportion of defenses that are bypassed at each attack stage is quantified by the attack stage, directly revealing the location of the defense weaknesses in the attack chain. This allows security operations personnel to prioritize the limited hardening resources to the attack stage with the highest penetration risk, rather than relying on experience-based judgment.
[0065] Blocking rate calculated using security device type and ATT&CK technology number as two-dimensional grouping keys and detection rate It can accurately pinpoint the protection gaps of specific device types when dealing with specific attack techniques. Compared to simply counting the overall blocking rate, it provides a more granular basis for evaluating device performance, enabling device rule optimization and strategy adjustment to be carried out on a specific technical category.
[0066] Area protection score By introducing alarm coverage coefficient The effectiveness of basic protection has been revised to ensure that, given the same blocking capabilities, network areas with stronger alarm detection capabilities receive appropriate score rewards, avoiding the neglect of detection capabilities when using blocking rate as the sole evaluation metric. Combining time-series score storage and alarms with continuously declining trends allows for the identification of protection capability degradation at an early stage. Regional importance weights have been introduced into remediation priority recommendations to ensure that remediation resources are allocated based on business importance rather than solely on technical indicators, aligning with the enterprise's actual security operation decision-making logic.
[0067] Example 2 In the method described in Example 1, the execution of the attack simulation task relies on pre-configured fixed execution scripts in the attack knowledge graph. Each ATT&CK technique number corresponds to one or more fixed attack use cases, and the blocking rate and detection rate calculated in steps 4 and 5 are based on the execution results of these fixed use cases. However, the security device's detection of attack behavior relies on pre-configured rule signatures or behavioral feature libraries, and the fixed scripts only cover specific known attack implementation variants. When an attacker modifies the attack payload by replacing encodings, obfuscating parameters, adjusting the execution order, or fragmenting the payload while adhering to the same ATT&CK technique primitives, the security device's rule signature may not match the modified attack behavior. In this case, the high blocking rate presented by the fixed use case execution results does not reflect the security device's rule tolerance for attack variants under the same ATT&CK technique, which may lead security managers to the erroneous judgment that all tactical stages have been effectively protected.
[0068] Based on Example 1, this implementation method introduces structured analysis of attack cases, application of systematic mutation operators, and statistics on mutation escape rates. It expands the attack cases under each ATT&CK technology number to a set of mutated cases covering multiple homogeneous mutation modes. On the basis of the multi-dimensional protection effectiveness evaluation framework of Example 1, it further quantifies the robustness of security device rules to attack behavior mutations under each ATT&CK technology dimension, thereby deepening the protection effectiveness conclusion from whether or not protection is provided to the protection failure under what mutation conditions and in what dimension.
[0069] At least one embodiment of the present invention discloses a multi-dimensional security protection effectiveness verification method based on the ATT&CK framework, which, in addition to the steps in Embodiment 1, further includes the following steps: Step 2-1: Perform structured parsing on the attack test case execution script to generate a test case variation space description structure.
[0070] The system retrieves the execution script corresponding to each CaseNode in the attack knowledge graph, performs structured parsing on each script, and extracts the set of mutable elements. The mutable elements fall into five categories: string constants (including URL paths, command-line argument values, user agent strings, etc.), encoding methods (including Base64, URL encoding, hexadecimal encoding, and other equivalent encoding forms), execution order (the sequential arrangement of multiple independent operation steps in the script), network protocol field values (including HTTP header field values, DNS query domain name formats, etc.), and payload fragmentation strategies (the method of splitting a single transmission into multiple fragmented transmissions).
[0071] Each mutable element is labeled with its element type tag and mutability range description, where the subscript is the element number. All mutable element descriptions for the same attack case are encapsulated together with the attack case's ATT&CK TechniqueID and CaseNode identifier to generate a test case mutation space description structure. ,in Index the use cases by number.
[0072] It should be noted that the extraction of the aforementioned mutable elements is based on static lexical analysis of the executed script. String literals, encoded function calls, loop transmission logic, and protocol construction statements in the script correspond to four types of elements: string constants, encoding methods, payload segmentation strategies, and network protocol field values, respectively. Execution order elements are identified through topology analysis of operation steps in the script that have no data dependencies. For use cases where the structure cannot be parsed (such as compiled binary payloads), the corresponding use case... If the set of mutable elements is marked as empty, this use case will only retain the original use case execution path in subsequent steps and will not generate a variant.
[0073] Step 2-2: Apply mutation operators based on the use case mutation space description structure to generate a set of mutated use cases.
[0074] Based on use case variation space description structure For each attack case, a predefined set of mutation operators is applied to generate a set of mutated cases. The mutation operator set includes the following five types of operators: String encoding replacement operator: Replaces mutable elements marked as string constants in the script with equivalent encoding forms, such as replacing plaintext URL paths with URL-encoded formats or hexadecimal escape sequences.
[0075] Parameter obfuscation operator: Inserts redundant characters into the command-line argument string that have no impact on the execution semantics or replaces the argument with an equivalent substitution, such as replacing / c with / / c, etc.
[0076] Execution order permutation operator: Performs full permutations and combinations of dependent operation steps marked as mutable elements of execution order class in the script to generate variants with different operation sequences but semantically equivalent.
[0077] Payload fragmentation operator: The single-transmission payload corresponding to the variable element marked as the payload fragmentation strategy class is split into multiple fragments according to the preset number of fragments and transmitted sequentially.
[0078] Protocol field randomization operator: Fill non-critical fields (i.e. fields that have no semantic impact on attack behavior) in elements marked as network protocol field value class with random valid values, such as randomizing the content of the User-Agent field in an HTTP request.
[0079] For each original use case, apply the five types of operators mentioned above independently, generating a preset number of operators for each type. One variant (default) Record the operator type used by each mutant. Specific mutation parameters and their corresponding ATT&CK technical numbers and original test case numbers are used to generate a mutation record. ,in This is the index for the variant number.
[0080] In this embodiment of the application, to avoid the combinatorial explosion between mutation operators causing the number of mutated test cases to exceed the processing capacity of the execution engine, Variants are generated individually according to operator type, without cross-combination between operators; that is, each variant corresponds to a single application result of a single mutation operator. The number of variants for each operator type is specified. The attack verification plan configuration can be customized.
[0081] Steps 2-3: Execute the original and variant use cases and collect the execution results.
[0082] Original attack cases and all variant cases Submit to the same attack simulation node as in step 2 of Example 1, and execute sequentially under the same target network environment, attacker node, target node, and security device configuration. Collect the same execution result data fields as in step 2 of Example 1 for each variant test case, including verification result (Result), block status (BlockStatus), alarm status (AlertStatus), and contextual information such as associated security device type (DeviceType), network zone (NetworkZone), and execution timestamp (ExecuteTime).
[0083] The execution result data of each variant test case is associated and stored with its corresponding original test case number, operator type and ATT&CK technology number to form a variant test case execution result dataset for subsequent aggregation analysis.
[0084] It should be noted that the logic for collecting and judging the execution results of mutated test cases is completely consistent with the processing of the original test cases in step 2 of Example 1. The verification results also adopt three states: effective, ineffective, and anomaly. Mutated test cases in the anomaly state are not included in the numerator or denominator in the subsequent mutation escape rate statistics.
[0085] Steps 2-4: Calculate the mutation escape rate by grouping according to the original use cases to obtain the rule robustness index vector.
[0086] The dataset of execution results for mutated test cases is grouped according to the original test case number. The following statistics are calculated for the execution results of each original test case and all its mutated test cases.
[0087] Overall escape rate Defined as the ratio of the number of valid validation results in a variant test case to the sum of the number of valid and invalid validation results in the variant test cases: ; in, Original use case The number of valid verification results out of all corresponding variant test cases. Original use case The number of test cases that failed the verification in the corresponding total number of variant test cases, excluding variant test cases with abnormal states in the denominator.
[0088] By mutation operator type Calculate the escape rate by category separately ,in: ; in, and These are the original use cases. In operator type The number of valid and invalid variant use cases.
[0089] Identify the mutation operator type with the highest classification escape rate in each original use case: ; Overall escape rate Escape rates of the five categories Co-encapsulation to generate original use cases Rule robustness index vector .
[0090] It should be noted that the above calculation of the escape rate uses the mutated test case groups of a single operator as an independent statistical unit. The escape rate results between different operator types are independent of each other. The most vulnerable variant dimension identified can directly pinpoint the variant category that causes the rule to fail, providing specific optimization directions for subsequent rule strengthening.
[0091] Steps 2-5: Cross-aggregate ATT&CK technology number and security device type to generate a multi-dimensional robustness distribution matrix.
[0092] The robustness index vector of rules According to ATT&CK technical number and types of safety equipment The two-dimensional grouping keys are cross-aggregated to calculate each Average escape rate of mutation under the combination: ; in, For associated security device type And ATT&CK's technical number is The original set of use cases, The number of use cases in this set. Index the original use cases. Original use case The overall escape rate of mutations.
[0093] In this embodiment of the application, in order to identify rule vulnerabilities masked by high blocking rates in fixed use cases, a surface security-substantial vulnerability determination rule is introduced: for each Combine and extract the blocking rate calculated in step 4 of Example 1. The combination is marked as surface-safe-substantially-vulnerable when the following conditions are met. ): ; in, The default value for the high blocking rate threshold is 70%. This is the high threshold for the average mutation escape rate, with a default value of 50%. Both of these thresholds can be customized in the attack verification plan configuration.
[0094] At the same time, statistics for each The distribution of classification escape rates for each mutation operator type in the combination is used to output the most vulnerable mutation operator type: ; All combinations , and Organize the data according to the two-dimensional coordinates of the ATT&CK tactical and technical layers to generate a multi-dimensional robust distribution matrix.
[0095] Steps 2-6: Extract typical escape mutation samples and generate a multi-dimensional security protection effectiveness verification report.
[0096] Based on the multi-dimensional robust distribution matrix, all labeled as of Combine and extract their corresponding Select from the valid variant test cases under the operator type. The top 3 mutation cases are used as typical escape mutation samples. The operator type, mutation parameters, and the values of the blocking status and alarm status fields during execution are recorded for each typical sample.
[0097] Integrating the above analysis results, a multi-dimensional security protection effectiveness verification report is generated. The multi-dimensional security protection effectiveness verification report adds the following content to the evaluation report in step 6 of Example 1: (1) Summary of Rule Robustness Analysis: According to Sort by highest to lowest and list all. of Each record contains a security device type identifier, ATT&CK technology number, tactical phase (obtained through a BELONGS_TO edge association query in the attack knowledge graph), and the original blocking rate calculated in Example 1. Average escape rate Most fragile mutation operator type And a characteristic description of typical escape variant samples.
[0098] (2) Visualization of multidimensional robustness distribution matrix: displayed in the form of heatmap The X-axis represents the type of security device, the Y-axis represents the type of ATT&CK attack technique, and the color depth of the heatmap corresponds to... The value, The combinations are distinguished by border markings.
[0099] In this embodiment of the application, in order to ensure that the repair priority recommendation can simultaneously reflect the original protection capability gap and the rule robustness gap, based on the calculation of the weighted risk score of each protection weakness in step 6 of embodiment one, the following is added: As an additional dimension, it is included in the repair priority ranking. The combination of these technologies, by adding the average mutation escape rate as a robustness penalty to the original weighted risk score, ensures that the combination of seemingly safe but substantially vulnerable equipment and technology receives a higher ranking in the repair priority list than combinations with the same original blocking rate gap but no robustness defects.
[0100] On the other hand, the present invention also proposes a verification system that is further extended based on the system described in Embodiment 1, by adding the following modules: The mutation space parsing module is used to obtain the execution script of each CaseNode in the attack knowledge graph. It extracts the set of mutable elements through static lexical analysis and topological analysis to generate a test case mutation space description structure. ; The variant test case generation module is used to describe the structure based on the test case mutation space. Five types of mutation operators are applied: string encoding replacement, parameter obfuscation, execution order substitution, payload fragmentation, and protocol field randomization, to generate mutant records. This forms a set of variant use cases; The mutation result acquisition module is used to submit the original test cases and mutated test cases to the attack simulation node for sequential execution, collect the execution result data fields of each mutated test case, and associate and store them with the original test case number, operator type, and ATT&CK technology number; The robustness metric calculation module is used to group the dataset of mutated test case execution results according to the original test case number and calculate the overall mutation escape rate. Classification escape rate and the most vulnerable mutation operator type Generate a rule robustness index vector ; The robustness matrix generation module is used to generate rule robustness index vectors. Cross-aggregation was performed based on ATT&CK technology designation and security device type to calculate the average escape rate. Performs surface security-substantial vulnerability determination, outputs a multi-dimensional robustness distribution matrix and Marking results; The escape sample extraction module is used to extract samples from samples labeled as of Typical escape mutation samples are extracted from the combination, and the robustness analysis summary and multi-dimensional robust distribution matrix visualization are output to the report generation module, and based on... Robust penalty items are recommended to be added to the repair priority.
[0101] By way of example, the system proposed in this invention may also include various features and combinations thereof in the method embodiments, which will not be elaborated here.
[0102] Building upon the multi-dimensional protection effectiveness evaluation framework of Embodiment 1, this invention extracts mutable elements through structured parsing of the attack case execution scripts. It then applies five types of mutation operators—string encoding replacement, parameter obfuscation, execution order substitution, payload fragmentation, and protocol field randomization—to expand the fixed attack cases under each ATT&CK technology number into a set of mutated cases covering multiple variations of the same attack pattern. This expansion broadens the statistical samples for blocking and detection rates from a single known implementation variant to multiple variant implementations under that technology number, enabling multi-dimensional protection evaluation to detect the tolerance boundaries of security device rules for subtle variations in attack behavior.
[0103] Mutant escape rate and escape rate The introduction of this feature allows for the quantification of rule robustness at the intersection of ATT&CK technology and security device type dimensions. The surface security-substantial vulnerability determination rule will determine the original blocking rate. With average escape rate Using the joint conditions as the basis for labeling, it can identify the device-technology combination that has a systematic failure of rule signature against polymorphic attacks under the condition of high blocking rate of fixed use cases, thereby overcoming the problem that high blocking rate of fixed use cases masks the vulnerability of rules and leads to the distortion of multi-dimensional evaluation conclusions.
[0104] The calculation further pinpoints rule vulnerabilities to specific mutation categories, enabling security operations personnel to optimize device rules for the most vulnerable mutation dimensions (such as code replacement or payload fragmentation), rather than performing a comprehensive, indiscriminate update of the rule base. The extraction of typical escape mutation samples provides concrete reference input for rule hardening, allowing remediation efforts to be validated based on real escape sample characteristics. A robustness penalty is overlaid in the remediation priority recommendation, ensuring that the combination of superficially secure but substantially vulnerable devices and technologies receives a remediation priority ranking that matches its actual risk level. This ensures that the allocation of security hardening resources simultaneously covers both the original protection capability gaps and the rule robustness gaps.
[0105] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.
Claims
1. A method for verifying the effectiveness of multi-dimensional security protection based on the ATT&CK framework, characterized in that, Includes the following steps: An attack knowledge graph is constructed, structured data from the ATT&CK framework is obtained, and attack cases are labeled according to four dimensions: attack stage, technology, sub-technology, and attack organization. The execution script, target platform, and precondition metadata of the attack cases are stored together with the multi-dimensional label information in the attack knowledge graph database. Execute attack simulations and collect execution results. Generate an attack task queue based on the attack verification plan. Drive the attack simulation nodes to perform simulated attack behaviors in the target network environment. Collect verification results, blocking status, alarm status, and associated security device type and network area for each attack case. Store these data in association with the case node in the attack knowledge graph. Aggregate the execution results data using the ATT&CK attack phase as the grouping key, and calculate the penetration risk index for each attack phase. ,in , To verify that the result represents the number of valid test cases, This represents the total number of use cases; The execution result data was cross-aggregated using security device type and ATT&CK technology number as two-dimensional grouping keys to calculate the blocking rate. and detection rate ; The execution result data is aggregated using the network region identifier as the grouping key, and the basic score of each network region is calculated. Alarm coverage coefficient and final protection score ; By integrating the above multi-dimensional statistical analysis results, a security protection effectiveness assessment report is generated.
2. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 1, wherein, The penetration risk indicator The calculation also includes: based on The range of values is used to mark the risk level of each attack phase. When marked as high risk, The time is marked as medium risk, when Time-marked as low risk; among them, test cases with abnormal verification results are included in the total number of test cases, but not in the remaining time-marked test cases. The numerator or denominator.
3. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 1, wherein, The blocking rate and detection rate The calculation also includes: and Safety equipment types and technology combinations that are all below the preset threshold are marked as serious protection blind spots.
4. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 1, wherein, The regional protection score Calculate as follows: ; ; ; in, This represents the number of test cases in this network area whose verification results are invalid. To verify that the result represents the number of valid test cases, To verify the number of test cases where the result was valid but triggered an alarm, This is the alarm weighting coefficient.
5. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 4 further includes a scoring alarm triggering step: when the protection score of a certain network area shows a downward trend for three consecutive days, or the daily score drop exceeds a preset threshold, an alarm notification is generated.
6. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 1, wherein, The remediation priority recommendations in the assessment report also include: calculating the weighted risk score of each protection weakness based on comprehensive penetration risk indicators, blocking rate gaps, network area ratings, and regional importance weights, and generating a priority list of remediation recommendations according to the scores from high to low.
7. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 1 further includes the step of mutating and expanding the attack cases: The execution scripts of each attack case in the attack knowledge graph are structured and parsed to extract five categories of mutable elements: string constants, encoding methods, execution order, network protocol field values, and payload partitioning strategies, generating a description structure of the test case mutation space. ; based on Five types of mutation operators—string encoding replacement, parameter obfuscation, execution order substitution, payload fragmentation, and protocol field randomization—are applied to generate mutant records. ; The original test cases and variant test cases are submitted to the attack simulation node for execution. The execution results are collected and stored in association with the original test case number, operator type, and ATT&CK technology number.
8. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 7 further includes the step of calculating the rule robustness index: The execution results of mutated test cases are grouped according to their original test case numbers, and the overall mutation escape rate is calculated. Classification escape rate by mutation operator type ,in , This represents the number of valid validation results in the variant test cases. The verification result is the number of failures; Identify the mutation operator type with the highest classification escape rate in each original use case. ,Will With each Co-encapsulation generates rule robustness index vector .
9. The multi-dimensional security protection effectiveness verification method based on the ATT&CK framework according to claim 8 further includes the step of generating a multi-dimensional robust distribution matrix: The robustness index vector of rules According to ATT&CK technical number and types of safety equipment Perform cross-aggregation to calculate the average mutation escape rate. ; When the blocking rate of a certain combination Not lower than the high threshold of blocking rate and Not lower than the high threshold of mutation escape rate When this condition is met, the combination is marked as a surface-safe-substantially-fragile state. ; Calculate the escape rate distribution of each mutation operator type in each combination, and output the most vulnerable mutation operator type. .
10. A multi-dimensional security protection effectiveness verification system based on the ATT&CK framework, characterized in that, include: The knowledge graph construction module is used to acquire structured data from the ATT&CK framework, label attack cases according to four dimensions: attack stage, technology, sub-technology, and attack organization, and generate and store the attack knowledge graph database. The attack simulation execution module is used to generate an attack task queue based on the attack verification plan, drive the attack simulation node to perform simulated attack behavior in the target network environment, collect the verification results, blocking status, alarm status and context information of each attack case, and associate and store them with the case node in the attack knowledge graph. The attack phase analysis module aggregates execution result data using the ATT&CK attack phase as the grouping key, and calculates the penetration risk index for each attack phase. and the percentage of each result type; The device protection analysis module is used to cross-aggregate the execution result data using security device type and ATT&CK technology number as two-dimensional grouping keys, and calculate the blocking rate of each device type against each attack technique. and detection rate Identify critical blind spots in protection; The regional scoring module is used to aggregate the execution result data with the network region identifier as the grouping key, calculate the basic score, alarm coverage coefficient and final protection score of each region, store the time-series scoring data and trigger scoring alarms; The report generation module integrates the outputs of the attack phase analysis module, device protection analysis module, and regional scoring module to generate a security protection effectiveness assessment report that includes ATT&CK attack phase protection effectiveness evaluation, security device type protection capability evaluation, network area comprehensive protection score and trend analysis, security protection risk ranking, and remediation priority recommendations.
