APT attack behavior-oriented portrait quality evaluation method and system, and terminal equipment

Abstract

The invention provides an APT (Advanced Persistent Threat) attack behavior-oriented portrait quality evaluation method and system and terminal equipment. The method comprises the following steps: constructing an evaluation index system covering multiple dimensions; setting a weight grade for each evaluation index of the evaluation index system; collecting evaluation index data of the to-be-evaluated portrait, and performing standardization processing on the collected data to obtain a standardization score; processing the missing index data, and performing compensation by adopting preset penalty score filling or weight recalculation according to the missing type; and calculating a comprehensive evaluation score based on the weight and the standardized score of each index, calculating a weight coverage rate of an effective coverage degree of quantitative index data, calculating an observation score reflecting effective observation index performance, and obtaining a quantitative evaluation result of the APT attack behavior portrait quality based on a calculation result. According to the method, comprehensive quantitative evaluation can be carried out on the APT attack behavior portrait, and security analysts are assisted in improving the credibility and practicability of a portrait system in attack traceability and defense decision making.

Description

Technical Field

[0001] This invention relates to the field of network information security technology, and in particular to a method, system, and terminal device for quality evaluation of APT attack behavior profiling. Background Technology

[0002] Advanced Persistent Threats (APTs) are cyberattacks that involve long-term, covert, and multi-stage infiltration of specific targets. Their attack chains typically span months or even years, exhibiting high levels of stealth and persistence. APT attack profiling, as a core technology for identifying, tracing, and defending against such attacks, integrates information such as behavioral characteristics, technical methods, and attack paths during the attack process. This provides security analysts with accurate attack situation awareness and decision support, and its quality directly determines the effectiveness and timeliness of the defense response.

[0003] Currently, the assessment of APT attack profiling quality largely relies on single-dimensional performance metrics and is primarily based on manual analysis. This analytical approach is highly subjective and lacks systematic, standardized, and multi-dimensional quantitative evaluation indicators, which hinders the effective comparison and deployment adaptation of different profiling methods or detection models.

[0004] Therefore, it is necessary to improve the existing methods for profiling and assessing APT attack behavior in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0005] To overcome the problems existing in related technologies, one of the objectives of this invention is to provide a quality evaluation method for APT attack behavior profiling. This method can achieve a comprehensive quantitative assessment of APT attack behavior profiling, assisting security analysts in improving the credibility and practicality of the profiling system in attack tracing and defense decision-making.

[0006] A method for quality assessment of APT attack behavior profiling includes: Construct an evaluation index system covering accuracy, completeness, timeliness, interpretability, generalization and robustness, operability, scalability and resource consumption; Weighting and grading are assigned to each evaluation indicator in the evaluation indicator system; Collect data on various evaluation indicators of the profile to be evaluated, and standardize the collected data to obtain standardized scores. For missing indicator data, compensation is carried out by filling with preset penalty scores or recalculating weights according to the type of missing data. The comprehensive evaluation score is calculated based on the weights and standardized scores of each indicator. The weighted coverage rate of the effective coverage of the quantitative indicator data is calculated, as well as the observation score reflecting the performance of the effective observation indicators. Based on the calculation results, the quantitative evaluation result of the APT attack behavior profile quality is obtained.

[0007] In a preferred embodiment of the present invention, the accuracy indicators include behavior recognition accuracy and attack phase matching degree; The integrity metrics include attack chain coverage and technical diversity. The timeliness indicators include profile generation latency and behavior lag rate; The interpretability metrics include attack path clarity and visualization score; The generalization and robustness metrics include the detection rate of unknown behaviors and the ability to resist disturbances; The operability indicators include the countermeasure recommendation rate and the ability to correlate events; The scalability metrics include concurrent processing capability and multi-source integration capability; The resource consumption metrics include average resource consumption and storage efficiency.

[0008] In a preferred embodiment of the present invention, the step of setting weight classifications for each evaluation index in the evaluation index system includes: The evaluation indicators in the evaluation indicator system are divided into multiple importance levels; the importance levels include at least four levels: core indicators, important indicators, medium indicators, and auxiliary indicators. Assign a preset weight value range to each importance level; Based on the relative importance of each evaluation indicator in the quality assessment of APT attack behavior profiling, it is assigned to a specific importance level and given a specific weight value under that level.

[0009] In a preferred embodiment of the present invention, the weight range of the core indicator is set to (0.10, 0.20). The weight range of the important indicators is set to (0.05, 0.10]. The weight range of the medium-weighted index is set to (0.03, 0.05). The weight range of the auxiliary indicator is set to (0.01, 0.03).

[0010] In a preferred embodiment of the present invention, the step of collecting data on various evaluation indicators of the portrait to be evaluated and standardizing the collected data to obtain a standardized score includes: Obtain the raw data of each evaluation indicator under the preset evaluation indicator system for the APT attack behavior profile to be evaluated; Determine the data type of each evaluation indicator; the data type includes at least proportional indicators, subjective rating indicators, inverse indicators, and count indicators. Based on the data type, the corresponding preset normalization function is called to process the original data, transforming the original data with different dimensions into dimensionless standardized scores, the standardized scores ranging from [0,1].

[0011] In a preferred embodiment of the present invention, the step of processing the original data by calling a corresponding preset normalization function according to the data type includes: The proportional index is linearly normalized and clipped to the [0,1] interval, as shown in the formula: s i =clip(x i (x, 0, 1), where x i For the original value of the indicator, s i For standardized scoring; the proportional indicators include behavior recognition accuracy, attack phase matching degree, attack chain coverage, unknown behavior detection rate, anti-disturbance capability, countermeasure recommendation rate, and storage efficiency; The subjective rating index is linearly mapped to the [0,1] interval, and the formula is: s i =(x i -1) / 4, where x i The raw score of the Likert scale; the subjective rating indicators include attack path clarity, visualization score, event correlation ability, and multi-source integration ability; The inverse indicator is normalized using the Logistic function, and the formula is: s i =exp(-x i / τ), where τ is the acceptable threshold for business, and its value is historical P75; the reverse indicators include profile generation latency, behavior lag rate, and average resource consumption. The count-type index is normalized using a saturation function, and the formula is: s i =x i / (x i +κ), where κ is a saturation constant with a historical value of P75–P90; the count-type indicators include technological diversity and concurrent processing capability.

[0012] In a preferred embodiment of the present invention, the step of processing the missing indicator data and compensating for the missing data by filling with a preset penalty score or recalculating the weights according to the type of missing data includes: The collected APT attack behavior profile multi-dimensional evaluation index data are screened to determine the specific types of missing index data. The missing types are divided into "should have but not measured" and "structure is not applicable". For missing indicator data of the "should have been but not measured" type, a preset penalty score is used to fill the missing data: based on the weight level of the missing indicator in the evaluation indicator system, the corresponding pre-set missing penalty score is applied. The missing data is filled in to obtain the compensated score for the indicator; For missing indicator data of the "structure inapplicable" type, weight recalculation compensation is adopted: the original weight of the "structure inapplicable" indicator is removed, the original weight of the remaining valid indicators is recalculated by normalization, and the new weight of each valid indicator is obtained. The new weight replaces the original weight in the subsequent evaluation calculation to achieve missing data compensation. By integrating the index scores after filling in the missing data with the effective index data after recalculating the weights, a complete index dataset without missing data is formed that can be used for comprehensive evaluation of APT attack behavior profiles.

[0013] In a preferred embodiment of the present invention, the step of calculating a comprehensive evaluation score, observation score, and weight coverage based on the weights and standardized scores of each indicator to obtain a quantitative evaluation result of the APT attack behavior profile quality includes: Obtain the processed basic data of the indicators: call the set of indicator weights after missing data processing and the set of indicator scores after standardization; Calculate the comprehensive evaluation score: Based on the weighted summation logic of indicator weights and standardized scores, each weight in the indicator weight set is multiplied by the corresponding score in the indicator score set and then summed. The comprehensive evaluation score is then obtained by converting to a percentage system, thereby achieving the overall quantification of the quality of APT attack behavior profiling. Calculate weight coverage: Calculate the total weights corresponding to the set of effective observation indicators to quantify the effective coverage of indicator data; Calculate the observation score: Based on the ratio of the weighted score of the effective observation indicators to the weight coverage, eliminate the interference of structurally inapplicable indicators on the score, and obtain a score that only reflects the performance of the effective observation indicators. Output quantitative evaluation results: Integrate and calculate the comprehensive evaluation score, weight coverage rate and observation score to form a multi-dimensional quantitative result that includes overall quality, indicator coverage and effective indicator performance, which is used for APT attack behavior profiling quality assessment and horizontal comparison.

[0014] The second objective of this invention is to provide a quality assessment system for profiling APT attack behavior, which is used to implement the quality assessment method for profiling APT attack behavior as described above.

[0015] A third objective of this invention is to provide a terminal device, comprising: a memory and a processor; The memory is used to store one or more computer instructions; The processor is used to execute one or more computer instructions to implement the APT attack behavior profiling quality evaluation method described above.

[0016] The beneficial effects of this invention are as follows: The present invention provides a quality evaluation method for APT attack behavior profiling. This method overcomes the limitations of existing technologies that rely solely on a single performance indicator (such as accuracy) by constructing a comprehensive indicator system covering eight dimensions: accuracy, completeness, timeliness, interpretability, generalization and robustness, operability, scalability, and resource consumption. At the same time, it considers accuracy, efficiency, practicality, and resource consumption, so that the evaluation results can comprehensively and realistically reflect the overall quality of the profiling system and avoid the one-sidedness of the evaluation. (2) By introducing a missing penalty score and a weight recalculation mechanism, this method can flexibly deal with the common problem of missing indicators in actual evaluation. This method can reasonably punish the missing indicators caused by system shortcomings and exempt the inapplicable indicators caused by architectural design. Thus, even in complex and imperfect real environments, it can still ensure the fairness and guiding significance of the evaluation results. (3) This method eliminates the influence of different indicator dimensions by designing customized standardized functions for different types of indicators; this allows different indicators to be compared on the same scale with weights, ensuring the scientific nature and fairness of the comprehensive evaluation results, and providing a unified comparison benchmark for different profiling systems or different versions of the same system. (4) This method also obtains a quantitative evaluation result of profiling quality by combining comprehensive evaluation score, observation score and weight coverage. The comprehensive evaluation score completes the missing indicators, realizing the overall quantification of APT attack behavior profiling quality. The indicator coverage and observation score can also reflect the performance of effective observation indicators.

[0017] This application also provides a system and terminal device for implementing the APT attack behavior profiling quality assessment method as described above, the system being able to perform the above method. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method for evaluating the quality of APT attack behavior profiling provided in an embodiment of the present invention; Figure 2 This is a flowchart of the standardization process for the collected data provided in an embodiment of the present invention; Figure 3 This is a logic diagram of the APT attack behavior profiling quality evaluation method provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of the evaluation index system provided in the embodiments of the present invention; Figure 5 This is an example diagram of the weighting of indicators for portrait quality assessment provided in an embodiment of the present invention. Detailed Implementation

[0019] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0020] Advanced Persistent Threats (APTs) are cyberattacks that involve long-term, covert, and multi-stage infiltration of specific targets. Their attack chains typically span months or even years, exhibiting high levels of stealth and persistence. APT attack profiling, as a core technology for identifying, tracing, and defending against such attacks, integrates information such as behavioral characteristics, technical methods, and attack paths during the attack process. This provides security analysts with accurate attack situation awareness and decision support, and its quality directly determines the effectiveness and timeliness of the defense response.

[0021] Currently, the assessment of APT attack profiling quality largely relies on single-dimensional performance metrics and is primarily based on manual analysis. This analytical approach is highly subjective and lacks systematic, standardized, and multi-dimensional quantitative evaluation indicators, which hinders the effective comparison and deployment adaptation of different profiling methods or detection models.

[0022] Based on this, this application provides a method for evaluating the quality of APT attack behavior profiling.

[0023] Example 1 like Figures 1-5 As shown in the figure, this embodiment provides a method for quality assessment of APT attack behavior profiling, including: S100. Construct an evaluation index system covering accuracy, completeness, timeliness, interpretability, generalization and robustness, operability, scalability and resource consumption. Specifically, the accuracy metrics include Behavior Identification Accuracy (BIA) and Attack Phase Matching (SMS). The integrity metrics include attack chain coverage (KCC) and technology diversity (TD). The timeliness indicators include profile generation latency (PL) and behavior lag rate (BLR). The interpretability metrics include Attack Path Clarity (APC) and Visualization Score (VS). The generalization and robustness metrics include Unknown Behavior Detection Rate (UBDR) and Perturbation Resistance (PR). The operational metrics include the response recommendation rate (CRR) and the event association capability (ICC). The scalability metrics include concurrent processing capability (CPT) and multi-source integration capability (MIS). The resource overhead metrics include average resource consumption (ARU) and storage efficiency (SE).

[0024] S200: Set weight levels for each evaluation indicator in the evaluation indicator system; Specifically, the steps are as follows: The evaluation indicators in the evaluation indicator system are divided into multiple importance levels; the importance levels include at least four levels: core indicators, important indicators, medium indicators, and auxiliary indicators. Assign a preset weight value range to each importance level; for example: The weight range of the core indicator is set to (0.10, 0.20); The weight range of the important indicators is set to (0.05, 0.10]. The weight range of the medium-weighted index is set to (0.03, 0.05). The weight range of the auxiliary indicator is set to (0.01, 0.03).

[0025] Based on the relative importance of each evaluation indicator in the quality assessment of APT attack behavior profiling, it is assigned to a specific importance level and given a specific weight value within that level. After assigning a specific weight value within that level, the specific weight values ​​of all evaluation indicators need to be summed to ensure that the sum is 1.

[0026] In practical applications, at least one of Behavior Recognition Accuracy (BIA), Attack Phase Matching (SMS), and Attack Chain Coverage (KCC) is assigned to the core indicator level. At least one of Technological Diversity (TD), Unknown Behavior Detection Rate (UBDR), Profile Generation Latency (PL), and Multi-Source Integration Capability (MIS) is assigned to the important indicator level. At least one of Attack Path Clarity (APC), Visualization Score (VS), Countermeasure Recommendation Rate (CRR), Event Correlation Capability (ICC), and Persistence Resistance (PR) is assigned to the intermediate indicator level. At least one of Behavior Lag Rate (BLR) and Concurrency Processing Capability (CPT) is assigned to the auxiliary indicator level.

[0027] S300. Collect data on various evaluation indicators of the profile to be evaluated, and standardize the collected data to obtain standardized scores. Specifically, the steps are as follows: S301. Obtain the raw data of each evaluation indicator of the APT attack behavior profile to be evaluated under the preset evaluation indicator system; S302. Determine the data type of each evaluation indicator; the data type includes at least proportional indicators, subjective scoring indicators, reverse indicators, and count indicators. S303. According to the data type, call the corresponding preset normalization function to process the original data, and convert the original data with different dimensions into dimensionless standardized scores. The value range of the standardized scores is [0,1].

[0028] Furthermore, the step of processing the original data by calling the corresponding preset normalization function according to the data type includes: The proportional index is linearly normalized and clipped to the [0,1] interval, as shown in the formula: s i =clip(x i (x, 0, 1), where x i For the original value of the indicator, s i For standardized scoring; the proportional indicators include behavior recognition accuracy, attack phase matching degree, attack chain coverage, unknown behavior detection rate, anti-disturbance capability, countermeasure recommendation rate, and storage efficiency; The subjective rating index is linearly mapped to the [0,1] interval, and the formula is as follows: s i =(x i -1) / 4, where x i The raw score of the Likert scale; the subjective rating indicators include attack path clarity, visualization score, event correlation ability, and multi-source integration ability; The inverse indicator is normalized using the Logistic function, and the formula is: s i =exp(-x i / τ), where τ is the acceptable threshold for business, and its value is historical P75; the reverse indicators include profile generation latency, behavior lag rate, and average resource consumption. The count-type index is normalized using a saturation function, and the formula is: s i =x i / (x i +κ), where κ is a saturation constant with a historical value of P75–P90; the count-type indicators include technological diversity and concurrent processing capability.

[0029] The following is a detailed description of how this application achieves the standardization of the collected data: By benchmarking, monitoring performance, and conducting expert reviews of a system, raw data for various evaluation indicators under a pre-defined evaluation indicator system are collected. For example, the raw data for some key indicators are shown in the table below: Then, based on the data characteristics of each indicator, its type is determined, and the corresponding preset normalization function is called for processing. The processing procedure is as follows: Proportional indicator processing: For indicators such as BIA and KCC, whose values ​​are already proportional, the clip function is directly called to clip them to the [0,1] range.

[0030] s_BIA=clip(0.95,0,1)=0.95 s_KCC=clip(0.75,0,1)=0.75 s_UBDR=clip(0.25,0,1)=0.25 Subjective rating index processing: For indicators such as APC and MIS based on a Likert five-point scale, a linear mapping formula s is used. i =(x i -1) / 4 is processed.

[0031] s_APC=(4-1) / 4=0.75 s_MIS=(3-1) / 4=0.50 Converse indicator handling: For indicators such as TD and CPT, where larger values ​​are generally better but the growth effect diminishes, a saturation function s is used. i =x i / (x i +κ) is normalized. The saturation constant κ is set according to the range of P75-P90 of historical data.

[0032] s_TD = 8 / (8+10) = 8 / 18 ≈ 0.44 s_CPT=150 / (150+300)=150 / 450≈0.33 After the above standardization process, the raw data of all indicators are transformed into standardized scores (s) in the interval [0,1]. i These scores eliminate the influence of dimensions, are additivity and comparability, and can be directly used for subsequent weighted summation to calculate the final comprehensive evaluation score of "System B".

[0033] By tailoring four different normalization functions for four different types of indicators (proportional, subjective, inverse, and count), the influence of different indicator dimensions is eliminated. This allows different indicators to be compared on the same scale with weights, ensuring the scientific nature and fairness of the comprehensive evaluation results and providing a unified comparison benchmark for different profiling systems or different versions of the same system. For example, "behavior recognition accuracy" and "profiling generation latency" can be weighted within the same mathematical framework, fundamentally guaranteeing the scientific nature and fairness of the comprehensive evaluation model. This embodiment does not use simple linear scaling for each indicator, but instead selects more intelligent functions based on the business meaning of the indicators. For example, exponential decay normalization is used for inverse indicators (latency, resource consumption), which can significantly amplify degradation phenomena exceeding the business threshold (τ), crucial for operation and maintenance scenarios that pursue low latency; saturation function normalization is used for count indicators (technical diversity, concurrency capability), which more objectively reflects the actual value of performance improvement. This processing makes the scoring results more reflective of the system's true business performance.

[0034] S400. Process the missing indicator data and compensate by filling with preset penalty scores or recalculating weights according to the missing type. The specific steps are as follows: The collected APT attack behavior profile multi-dimensional evaluation index data are screened to determine the specific types of missing index data. The missing types are divided into "should have but not measured" and "structure is not applicable". For missing indicator data of the "should have been but not measured" type, a preset penalty score is used to fill the missing data: based on the weight level of the missing indicator in the evaluation indicator system, the corresponding pre-set missing penalty score is applied. The missing data is filled in to obtain the compensated score for the indicator; For missing indicator data of the "structure inapplicable" type, weight recalculation compensation is adopted: the original weight of the "structure inapplicable" indicator is removed, the original weight of the remaining valid indicators is recalculated by normalization, and the new weight of each valid indicator is obtained. The new weight replaces the original weight in the subsequent evaluation calculation to achieve missing data compensation. By integrating the index scores after filling in the missing data with the effective index data after recalculating the weights, a complete index dataset without missing data is formed that can be used for comprehensive evaluation of APT attack behavior profiles.

[0035] In one implementation, a comprehensive screening of 16 indicators from 5 APT attack profiles was conducted, checking data collection records and system function documents one by one to identify the specific types of missing indicators ("should be present but not measured" or "structure is not applicable"). The screening results are shown in the table below: Based on the pre-defined rules of the evaluation index system, the weighting of each missing index is as follows: TD (Technology Diversity): Weighted as a key indicator (High), with an initial weight of 0.08; ICC (Event Correlation Capability): The weighting is based on the core indicator, with an initial weight of 0.05.

[0036] Based on the binding rules of weighted grading and penalty scores (core indicator penalty score 0, important indicator penalty score 0.15, medium indicator penalty score 0.3, auxiliary indicator penalty score 0.5), the missing data is filled in: TD missing data imputation: The penalty score of 0.15 corresponding to the important indicator is used to obtain the TD-compensated score. ; ICC Missing Data Imputation: The penalty score corresponding to the core indicator is 0, and the score is obtained after compensation for the ICC. .

[0037] After filling, the two portraits with missing TD and the one portrait with missing ICC all obtained valid scores for this indicator, with no data gaps.

[0038] Then remove the original weights of the "Structure Inapplicable" indicator: Visualization score (VS) is an auxiliary indicator (Low) with an original weight of 0.05. For the three profiles with missing VS, the original weight of VS is first removed, and the sum of the original weights of the remaining 15 effective indicators is S=1-0.05=0.95.

[0039] Following the logic of "new weight = original weight ÷ sum of remaining effective indicator weights", the weights of the 15 indicators were recalculated. The recalculation results of some core indicators are shown in the table below: After recalculation, the sum of the new weights of the 15 effective indicators is 1, ensuring the integrity and logical consistency of the weighting system.

[0040] The filled indicator scores (TD=0.15, ICC=0) are integrated with the effective indicator data after recalculating the weights in step 300 (such as the new weight of BIA 0.1895, the new weight of SMS 0.1263, etc.), while retaining the original missing records and processing traces (such as "VS-structure not applicable-weight removal", "TD-should exist but not measured-penalty score filling 0.15"). Finally, five complete indicator datasets without missing data are formed, which can be directly used for subsequent comprehensive evaluation score calculation.

[0041] By differentiating between two types of deficiencies—"should have but not measured" and "structural inapplicability"—and treating them accordingly, this application achieves refined fairness. Penalizing deficiencies caused by system capability shortcomings or insufficient testing (such as TD and VS) prevents "filler" deficiencies; while exempting inherent inapplicability due to reasonable architectural design (such as ICC) avoids "punishment without fault." This mechanism ensures that when comparing profiling systems with different system architectures and capability focuses, the evaluation results maintain a high degree of fairness and credibility.

[0042] This application transforms missing data handling from a temporary, subjective decision-making process into a structured, evidence-based procedure. Penalty scores are linked to weighted levels, and weight recalculation follows strict mathematical rules, making the process highly transparent, enhancing the interpretability of evaluation results, and facilitating analysts' precise identification of areas for system improvement.

[0043] S500 calculates a comprehensive evaluation score based on the weights and standardized scores of each indicator, calculates the weighted coverage rate of the effective coverage of quantitative indicator data, and calculates the observation score reflecting the performance of effective observation indicators. Based on the calculation results, a quantitative evaluation result of the quality of the APT attack behavior profile is obtained.

[0044] The specific steps are as follows: Obtain the processed basic data of the indicators: call the set of indicator weights after missing data processing and the set of indicator scores after standardization; Calculate the comprehensive evaluation score: Based on the weighted summation logic of indicator weights and standardized scores, each weight in the indicator weight set is multiplied by the corresponding score in the indicator score set and then summed. The comprehensive evaluation score is then obtained by converting to a percentage system, thereby achieving the overall quantification of the quality of APT attack behavior profiling. Calculate weight coverage: Calculate the total weights corresponding to the set of effective observation indicators to quantify the effective coverage of indicator data; Calculate the observation score: Based on the ratio of the weighted score of the effective observation indicators to the weight coverage, eliminate the interference of structurally inapplicable indicators on the score, and obtain a score that only reflects the performance of the effective observation indicators. Output quantitative evaluation results: Integrate and calculate the comprehensive evaluation score, weight coverage rate and observation score to form a multi-dimensional quantitative result that includes overall quality, indicator coverage and effective indicator performance, which is used for APT attack behavior profiling quality assessment and horizontal comparison.

[0045] In one implementation, the step is as follows: The basic data of the indicators after standardization in step S300 and missing data processing in step S400 are retrieved, including the processed weight set W' and standardized score set S' of 16 indicators. The core data is shown in the table below (only the key indicators are listed, the complete data contains 16 items).

[0046] Note: P2 / P3 were removed from their original weight of 0.05 due to "structure inapplicability" in VS, and the weights of the remaining 15 indicators were renormalized.

[0047] (Recalculation formula: The TD of P3 is "should have been but not measured", so the penalty score corresponding to the important indicator is 0.15.

[0048] Calculate the overall evaluation score: Using the logic of "weighted summation + percentage conversion", the calculation formula is as follows: ; Where (n is the total number of indicators, P1=16, P2 / P3=15; For the processed weights, (Standardized score).

[0049] Example of calculating the comprehensive evaluation score for each portrait (taking P3 as an example): BIA contribution: 0.1895 × 0.85 ≈ 0.1611; SMS contribution: 0.1263 × 0.79 ≈ 0.1008; KCC contribution: 0.1263 × 0.73 ≈ 0.0922; TD contribution: 0.0842 × 0.15 ≈ 0.0126; ICC contribution: 0.0526 × 0.62 ≈ 0.0326; The cumulative contribution of the remaining 10 indicators is approximately 0.2497.

[0050] Weighted summation: 0.1611 + 0.1008 + 0.0922 + 0.0126 + 0.0326 + 0.2497 ≈ 0.649; P3 overall evaluation score: 100 × 0.649 = 64.9 points.

[0051] Summary of overall evaluation scores for the three portraits: P1 (No missing parts): Score = 78.3 (Good grade: 70≤78.3≤84); P2 (VS missing only): Score = 72.5 (Good grade); P3 (VS+TD missing): Score = 64.9 (Level of improvement needed: 50≤64.9≤69).

[0052] Then, using the logic of "summing the weights of effective observation indicators", the calculation formula is as follows: ; O represents the set of valid observation indicators, including those collected normally and those filled in during the missing period, excluding NA indicators.

[0053] Calculation of coverage weight for each portrait: P1 (without NA indicator): C = 0.18 + 0.12 + 0.12 + ... + 0.03 = 1.0 (100% coverage) P2 (excluding VS): C = 1.0 - 0.05 = 0.95 (95% coverage) P3 (excluding VS): C=0.95 (95% coverage, TD is Missing filler, which is a valid observation indicator) Step 400: Calculate the observation score.

[0054] The calculation formula uses the logic of "weighted sum of effective observation indicators ÷ weighted coverage rate + percentage conversion", and is as follows: ; Example of scoring calculation for each portrait (taking P3 as an example): Weighted sum of effective observation indicators: (Consistent with the weighted sum of the overall scores, as there are no missing indicators other than NA). P3 observation score: 100×(0.649÷0.95)≈68.3 points.

[0055] Summary of observation scores for 3 portraits: P1: Score obs =100×(0.783÷1.0)=78.3 points (consistent with the overall score); P2: Score obs =100×(0.689÷0.95)≈72.5 points (consistent with the overall score); P3: Score obs =68.3 points (higher than the overall score of 64.9 points, after removing the dilution effect of VS weight).

[0056] Finally, the three core data points from the three profiles will be integrated to form a multi-dimensional quantitative results report for output.

[0057] This application simultaneously outputs a comprehensive evaluation score, weight coverage rate, and observation score, forming a "three-dimensional" quantitative result: the comprehensive score reflects the overall quality of the profile (including missing value penalties and weight adjustments), the weight coverage rate quantifies the effective coverage of indicators, and the observation score focuses on the true performance of effective indicators, avoiding misjudgments caused by a single score and achieving a three-dimensional portrayal of profile quality. Through weight recalculation and observation score, the core performance gap between the lightweight system and the full-featured system is accurately reflected, rather than a false gap caused by functional simplification, ensuring horizontal comparability between profile systems with different architectures.

[0058] Example 2 This embodiment provides a quality assessment system for profiling APT attack behavior, used to implement the quality assessment method for profiling APT attack behavior as described above.

[0059] This system automates the entire process from indicator construction to result output. Each module performs its specific function, eliminating inconsistencies and subjectivity inherent in manual operations. This ensures that every evaluation follows a unified and standardized process, improving the comparability and repeatability of evaluation results.

[0060] By implementing the methods described above, the system automatically performs attribution analysis on quality issues through a combination of observation scores and weighted coverage, as well as radar chart visualization. As shown in the example, the system can intuitively reveal whether the score reduction is due to "functional deficiencies" or "performance inadequacies," and provides specific optimization directions through an improvement suggestion generator, upgrading the assessment from a simple "score" to "diagnosis and guidance."

[0061] Example 3 This embodiment provides a terminal device, including: a memory and a processor; The memory is used to store one or more computer instructions; The processor is used to execute one or more computer instructions to implement the above-described method for evaluating the quality of APT attack behavior profiling.

[0062] Specifically, the terminal device's memory is used to store one or more computer instructions; the terminal device's processor is connected to the memory and is used to execute the one or more computer instructions; the communication interface is used to interact with external systems and receive APT attack behavior profile data; it also includes a display unit for displaying evaluation results and a visualization interface.

[0063] When the processor executes the computer instructions, it works according to the following process: acquiring APT attack behavior profile data to be evaluated through the communication interface; Retrieve the indicator system configuration and weight settings stored in the memory; Perform data standardization and missing data handling algorithms; Perform comprehensive evaluation calculations; The display unit outputs multidimensional quantitative results including comprehensive evaluation score, weight coverage, and observation score.

[0064] This invention enables the evaluation scheme to be rapidly deployed as a standalone product in various network security management environments by implementing the evaluation method within a terminal device containing memory and a processor. The processor ensures efficient execution of complex calculations, overcoming the efficiency bottlenecks of manual calculations or distributed script processing, and significantly improving the execution speed and stability of evaluation tasks. This terminal device can be integrated with actual network security monitoring platforms, log management systems, or SIEM (Security Information and Event Management Systems) via a communication interface to directly acquire real APT attack behavior stream data for real-time or near-real-time quality assessment, enhancing its practical application capabilities in real network environments.

[0065] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings. In the description of this application, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is usually based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0066] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for quality assessment of APT attack behavior profiling, characterized in that, include: Construct an evaluation index system that includes accuracy evaluation index, completeness evaluation index, timeliness evaluation index, interpretability evaluation index, generalization and robustness evaluation index, operability evaluation index, scalability evaluation index and resource consumption evaluation index; Weighting and grading are assigned to each evaluation indicator in the evaluation indicator system; Collect the data required for each evaluation indicator of the profile to be evaluated, and standardize the collected data to obtain standardized scores; For missing indicator data, compensation is carried out by filling with preset penalty scores or recalculating weights according to the type of missing data. The comprehensive evaluation score is calculated based on the weights and standardized scores of each indicator. The weighted coverage rate, which quantifies the effective coverage of indicator data, is also calculated. The observation score, which reflects the performance of effective observation indicators, is also calculated. Based on the comprehensive evaluation score, weighted coverage rate, and observation score, the quantitative evaluation result of the APT attack behavior profile quality is obtained.

2. The method for quality assessment of APT attack behavior profiling according to claim 1, characterized in that: The accuracy evaluation metrics include behavior recognition accuracy and attack phase matching degree; The integrity evaluation metrics include attack chain coverage and technical diversity. The timeliness evaluation indicators include profile generation latency and behavior lag rate; The interpretability evaluation metrics include attack path clarity and visualization score; The generalization and robustness evaluation metrics include the unknown behavior detection rate and the ability to resist disturbances; The operability evaluation indicators include the countermeasure recommendation rate and the event correlation capability; The scalability evaluation metrics include concurrent processing capability and multi-source integration capability; The resource consumption evaluation metrics include average resource consumption and storage efficiency.

3. The method for quality assessment of APT attack behavior profiling according to claim 1, characterized in that: The process of assigning weight levels to each evaluation indicator in the evaluation indicator system includes: The evaluation indicators in the evaluation indicator system are divided into multiple importance levels; the importance levels include at least four levels: core indicators, important indicators, medium indicators, and auxiliary indicators. Assign a preset weight value range to each importance level; Based on the relative importance of each evaluation indicator in the quality assessment of APT attack behavior profiling, it is assigned to a specific importance level and given a specific weight value under that level.

4. The method for quality assessment of APT attack behavior profiling according to claim 3, characterized in that: The weight range of the core indicator is set to (0.10, 0.20); The weight range of the important indicators is set to (0.05, 0.10]. The weight range of the medium-weighted index is set to (0.03, 0.05). The weight range of the auxiliary indicator is set to (0.01, 0.03).

5. The method for quality assessment of APT attack behavior profiling according to any one of claims 2-4, characterized in that: The process involves collecting data on various evaluation indicators for the profile to be evaluated, and then standardizing the collected data to obtain standardized scores, including: Obtain the raw data of each evaluation indicator under the preset evaluation indicator system for the APT attack behavior profile to be evaluated; Determine the data type of each evaluation indicator; the data type includes at least proportional indicators, subjective rating indicators, inverse indicators, and count indicators. Based on the data type, the corresponding preset normalization function is called to process the original data, transforming the original data with different dimensions into dimensionless standardized scores, the standardized scores ranging from [0,1].

6. The method for quality assessment of APT attack behavior profiling according to claim 5, characterized in that: The step of processing the original data by calling the corresponding preset normalization function according to the data type includes: The proportional index is linearly normalized and clipped to the [0,1] interval, as shown in the formula: s i =clip(x i (x, 0, 1), where x i For the original value of the indicator, s i For standardized scoring; the proportional indicators include behavior recognition accuracy, attack phase matching degree, attack chain coverage, unknown behavior detection rate, anti-disturbance capability, countermeasure recommendation rate, and storage efficiency; The subjective rating index is linearly mapped to the [0,1] interval, and the formula is as follows: s i =(x i -1) / 4, where x i The raw score of the Likert scale; the subjective rating indicators include attack path clarity, visualization score, event correlation ability, and multi-source integration ability; The inverse indicator is normalized using the Logistic function, and the formula is: s i =exp(-x i / τ), where τ is the acceptable threshold for business, and its value is historical P75; the reverse indicators include profile generation latency, behavior lag rate, and average resource consumption. The count-type index is normalized using a saturation function, and the formula is: s i =x i / (x i +κ), where κ is a saturation constant with a historical value of P75–P90; the count-type indicators include technological diversity and concurrent processing capability.

7. The method for quality assessment of APT attack behavior profiling according to any one of claims 1-4, characterized in that: The process of processing missing indicator data, and compensating for the missing data by using preset penalty scores or weight recalculation according to the type of missing data, includes: The collected APT attack behavior profile multi-dimensional evaluation index data are screened to determine the specific types of missing index data. The missing types are divided into "should have but not measured" and "structure is not applicable". For missing indicator data of the "should have been but not measured" type, a preset penalty score is used to fill the missing data: based on the weight level of the missing indicator in the evaluation indicator system, the corresponding pre-set missing penalty score is applied. The missing data is filled in to obtain the compensated score for the indicator; For missing indicator data of the "structure inapplicable" type, weight recalculation compensation is adopted: the original weight of the "structure inapplicable" indicator is removed, the original weight of the remaining valid indicators is recalculated by normalization, and the new weight of each valid indicator is obtained. The new weight replaces the original weight in the subsequent evaluation calculation. Integrate the indicator scores after filling in the missing data with the effective indicator data after recalculating the weights to form an indicator dataset.

8. The method for quality assessment of APT attack behavior profiling according to any one of claims 1-4, characterized in that: The quantitative evaluation results of the APT attack behavior profile quality are obtained by calculating the comprehensive evaluation score, observation score, and weight coverage based on the weights and standardized scores of each indicator, including: Obtain the processed basic data of the indicators: call the set of indicator weights after missing data processing and the set of indicator scores after standardization; Calculate the comprehensive evaluation score: Based on the weighted summation logic of indicator weights and standardized scores, multiply each weight in the indicator weight set with the corresponding score in the indicator score set one by one, and then sum them up. Finally, convert the result to a percentage system to obtain the comprehensive evaluation score. Calculate weight coverage: Calculate the total weights corresponding to the set of effective observation indicators to quantify the effective coverage of indicator data; Calculate the observation score: Based on the ratio of the weighted score of the effective observation indicators to the weight coverage, the interference of structurally inapplicable indicators on the score is eliminated, and the score that only reflects the performance of the effective observation indicators is obtained. Output quantitative evaluation results: Integrate and calculate the comprehensive evaluation score, weight coverage rate and observation score to form a multi-dimensional quantitative result that includes overall quality, indicator coverage and performance of effective indicators.

9. A quality assessment system for profiling APT attack behavior, characterized in that: Used to implement the APT attack behavior profiling quality evaluation method as described in any one of claims 1-8.

10. A terminal device, characterized in that, include: Memory, processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement as claimed in claim 1.

8. A method for evaluating the quality of APT attack behavior profiling as described in any one of the following.

Images