Network security situation awareness system capability evaluation method, system, device and medium

Abstract

The application provides a network security situation awareness system capability evaluation method, system, device and medium, and relates to the field of network security, comprising the following steps: determining an evaluation purpose based on a network security situation awareness system to be evaluated, and determining a test scene according to the evaluation purpose; constructing a test network environment, selecting a simulation attack tool and an attack tactic, selecting a network device to be attacked and a data type tool based on the test scene, generating a test process, and collecting test data in the test process; constructing a multi-dimensional network security situation awareness evaluation index based on the test scene; comparing and analyzing test data, awareness data obtained by the network security situation awareness system to be evaluated and output awareness results by using an artificial intelligence large language model based on the multi-dimensional network security situation awareness evaluation index; and obtaining an evaluation result. The application realizes the evaluation of the multi-aspect capability of the network security situation awareness system, and improves the systematized network security evaluation technical level.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a method, system, device, and medium for evaluating the capabilities of a network security situation awareness system. Background Technology

[0002] With the rapid development of next-generation information technologies such as 5G, big data, and artificial intelligence, and the arrival of the era of the Internet of Things, the sources and methods of cybersecurity threats are constantly evolving, and the complexity and harm of cybersecurity are becoming increasingly prominent, necessitating the establishment of a dynamic and comprehensive protection concept. Cybersecurity situational awareness is a technology that detects, extracts, understands, assesses, and predicts the future of security elements affecting the network situation in a large-scale network environment. As a real-time guardian of cybersecurity, the cybersecurity situational awareness system is the main means of achieving "all-weather, all-round perception of the cybersecurity situation." Currently, there are numerous cybersecurity situational awareness systems based on various technologies, and their cybersecurity situational awareness capabilities vary considerably.

[0003] Currently, mainstream evaluation methods generally adopt virtualization and other technologies in terms of environment construction to flexibly build attack and defense environments. In terms of evaluation, they each have their own focus, and there is no automated orchestration capability to automatically build evaluation environments and call up testing tools according to the test scenario. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, system, device and medium for evaluating the capabilities of a network security situation awareness system, thereby evaluating the capabilities of the network security situation awareness system in areas such as network security attack analysis, asset risk analysis, abnormal behavior analysis and situation display, so as to improve the level of systematic network security evaluation technology.

[0005] This application provides the following technical solution: a method for evaluating the capabilities of a network security situation awareness system, comprising:

[0006] Based on the network security situation awareness system to be evaluated, the evaluation objective is determined, and test scenarios are selected from a preset scenario library according to the evaluation objective;

[0007] Based on the test scenario, a test network environment is constructed, simulated attack tools are selected from a pre-constructed attack toolset and corresponding attack tactics are determined, network devices to be attacked are selected from a pre-constructed target toolset, and data tools are selected from a pre-constructed data toolset; wherein, the attack toolset, the target toolset, and the data toolset are pre-constructed based on historical network security situation awareness and test data, respectively.

[0008] Based on the test network environment, the simulated attack tools and corresponding attack tactics, the network devices to be attacked and the data tools, a test process is generated and executed, and test data is collected during the test process.

[0009] A multi-dimensional network security situation awareness assessment index based on test scenarios is constructed. Based on the multi-dimensional network security situation awareness assessment index, the test data is compared and analyzed with the perception data and output perception results obtained by the network security situation awareness system to be evaluated through an artificial intelligence large language model to obtain the evaluation results.

[0010] According to one embodiment of this application, it further includes: containerizing and deploying operating system images, target machine tool images, attack tool images, and data tool images, and managing the operating system images, target machine tool images, attack tool images, and data tool images in the form of container images.

[0011] According to one embodiment of this application, it further includes:

[0012] Based on the comparison results between the test data and the perception data obtained by the network security situation awareness system to be evaluated, the evaluation index scores of each network security situation awareness evaluation index are calculated.

[0013] Based on the degree of impact of each network security situation awareness assessment indicator on network security, weights are designed for each network security situation awareness assessment indicator. The assessment results are calculated based on the assessment indicator scores and corresponding weights of each network security situation awareness assessment indicator.

[0014] According to one embodiment of this application, the multi-dimensional network security situation awareness assessment indicators include: the ability to generate a visualized global network topology, the completeness of network traffic data collection, the completeness of log data collection, the accuracy of open port and vulnerability information, the accuracy of threat intelligence, the ability to identify online assets, the accuracy of alarm push notifications, and the ability to reconstruct the attack chain through multi-dimensional related events.

[0015] According to one embodiment of this application, the attack toolset includes vulnerability exploitation attack tools, malicious code attack tools, web application attack tools, data theft attack tools, malicious email attack tools, comprehensive orchestration attack tools, malicious address access tools, inappropriate content access tools, and malicious behavior access tools.

[0016] According to one embodiment of this application, the method further includes: after completing the construction of the test network environment and determining the network device to be attacked, simulating the user's online network activity behavior to generate real network traffic by simulating the user's operation behavior on the network application.

[0017] According to one embodiment of this application, it further includes:

[0018] In the test network environment, real-time attack data streams and offline attack data streams are started in parallel. The traffic of the real-time attack data stream is mirrored on the network devices along the attack path and forwarded to the network security situation awareness system to be evaluated, thus completing the collection of real-time attack traffic data.

[0019] The offline attack data stream is labeled, denoised, and normalized to generate a standardized test dataset, which is then sent to the network security situation awareness system to be evaluated.

[0020] This application also provides a network security situation awareness system capability evaluation system, including:

[0021] The scenario determination module is used to determine the evaluation purpose based on the network security situation awareness system to be evaluated, and to determine the test scenario from the preset scenario library according to the evaluation purpose;

[0022] The environment construction and management module is used to construct a test network environment based on the test scenario, select simulated attack tools from a pre-built attack toolset and determine the corresponding attack tactics, select network devices to be attacked from a pre-built target toolset, and select data tools from a pre-built data toolset; wherein the attack toolset, the target toolset, and the data toolset are pre-built based on historical network security situation awareness and test data, respectively.

[0023] The testing module is used to generate a test process based on the test network environment, the simulated attack tool and corresponding attack tactics, the network device to be attacked and the data tool, and then execute the test process and collect test data during the test process.

[0024] The evaluation module is used to construct a multi-dimensional network security situation awareness evaluation index based on the test scenario. Based on the multi-dimensional network security situation awareness evaluation index, the test data is compared and analyzed with the perception data and output perception results obtained by the network security situation awareness system to be evaluated through an artificial intelligence large language model to obtain the evaluation result.

[0025] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described network security situation awareness system capability evaluation method.

[0026] This application also provides a computer-readable storage medium storing a computer program that performs the above-described network security situation awareness system capability assessment method.

[0027] Compared with existing technologies, the beneficial effects achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: The embodiments of this invention comprehensively evaluate the capabilities of a network security situation awareness platform in areas such as network security attacks, asset risk analysis, and abnormal behavior analysis. These embodiments construct a network security situation awareness product detection environment, employing virtualization and containerization technologies to build various target machines; integrate multiple attack tools and datasets capable of simulating mainstream attack scenarios; create a multi-dimensional network security situation awareness capability evaluation index system; and utilize large-scale artificial intelligence models to analyze the results of network security situation awareness capabilities. This allows for a comprehensive and effective evaluation of the situation awareness product's capabilities in areas such as network attacks, asset risks, and abnormal behavior analysis. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a first schematic diagram of the process for evaluating the network security situation awareness system capability according to an embodiment of the present invention;

[0030] Figure 2 This is a second schematic diagram of the process for evaluating the network security situation awareness system capability according to an embodiment of the present invention;

[0031] Figure 3 This is a first schematic diagram of the network security situation awareness system capability evaluation system structure according to an embodiment of the present invention;

[0032] Figure 4 This is a second schematic diagram of the network security situation awareness system capability evaluation system structure according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the structure of the computer device of the present invention. Detailed Implementation

[0034] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0035] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] The terms used in the embodiments of this invention are explained as follows:

[0037] Threat: A potential factor of an undesirable event that could harm a system or organization.

[0038] Threat information: Evidence-based knowledge, including context, attack mechanisms, attack indicators, and potential impact. Used to describe existing or potential threats, enabling threat response and prevention.

[0039] Network security situation awareness: By collecting data such as network traffic, asset information, logs, vulnerability information, alarm information, and threat information, analyzing and processing factors such as network behavior and user behavior, we can grasp the network security status, predict network security trends, and carry out activities such as display, detection, and early warning.

[0040] like Figure 1 As shown, this embodiment of the invention provides a method for evaluating the capabilities of a network security situation awareness system, including:

[0041] S101. Based on the network security situation awareness system to be evaluated, determine the evaluation objective, and determine the test scenario from the preset scenario library according to the evaluation objective;

[0042] S102. Based on the test scenario, construct a test network environment, select simulated attack tools from a pre-built attack toolset and determine the corresponding attack tactics, select network devices to be attacked from a pre-built target toolset, and select data tools from a pre-built data toolset; wherein, the attack toolset, the target toolset, and the data toolset are pre-built based on historical network security situation awareness and test data, respectively.

[0043] S103. Based on the test network environment, the simulated attack tool and corresponding attack tactics, the network device to be attacked and the data tool, generate a test process, execute the test process, and collect test data during the test process;

[0044] S104. Construct a multi-dimensional network security situation awareness evaluation index based on the test scenario. Based on the multi-dimensional network security situation awareness evaluation index, compare and analyze the test data with the perception data and output perception results obtained by the network security situation awareness system to be evaluated through an artificial intelligence large language model to obtain the evaluation results.

[0045] This invention provides a method for evaluating the comprehensive capabilities of a network security situation awareness platform in network security attack analysis, asset risk analysis, abnormal behavior analysis, and situational awareness. This invention constructs a network security situation awareness product testing environment, establishes target machines simulating mainstream attack scenarios, and develops verification and analysis tools for the situation awareness system's ability to integrate network attacks, asset risks, and abnormal behaviors into security event analysis. This has resulted in a relatively systematic evaluation method, toolset, and system, thereby improving the level of systematic network security evaluation technology.

[0046] In one embodiment of the present invention, the method further includes: containerizing and deploying operating system images, target machine tool images, attack tool images, and data tool images, and managing the operating system images, target machine tool images, attack tool images, and data tool images in the form of container images.

[0047] In this embodiment, all network topology, operating system, target machine services, attack tools, and test data used in the evaluation network environment are made into Docker / OCI images (i.e., container images). These images are then placed into a private image repository according to their categories, with unified versions, unified pulls, and unified upgrades. In practice, each image can be tagged to ensure 100% reproducibility of the same evaluation scenario. The evaluation platform uses the `docker pull` command to pull images in seconds, enabling rapid distribution. Furthermore, a single command using `docker-compose` or Helm assembles the "OS + target machine + attack tools + data" into a complete topology, enabling rapid scenario assembly.

[0048] In one embodiment of the present invention, the method further includes: calculating the evaluation index scores of each network security situation awareness evaluation index based on the comparison results between the test data and the perception data obtained by the network security situation awareness system to be evaluated; designing weights for each network security situation awareness evaluation index based on the degree of influence of each network security situation awareness evaluation index on network security; and calculating the evaluation result based on the evaluation index scores and corresponding weights of each network security situation awareness evaluation index.

[0049] In specific implementation, such as Figure 2 As shown in this embodiment, a method for evaluating the capabilities of a network security situation awareness system includes the following steps:

[0050] Based on the evaluation objectives, the required test scenarios are determined. Based on these scenarios, the algorithm automatically constructs the complex simulated network environment needed for the test, selects simulated attack tools and tactics, and builds the network devices to be attacked. It can quickly and automatically plan, deploy, expand, release, and schedule resources, allowing evaluation personnel to manage resources. Management of operating system images, target machine images, attack tool images, and data images is achieved using container images. The operating system images support mainstream operating system categories to meet the compatibility requirements of situational awareness probe deployment, target machine deployment, and attack tool deployment. Attack tools include: vulnerability exploitation tools, malware attack tools, web application attack tools, data theft attack tools, malicious email attack tools, comprehensive orchestration attack tools, malicious address access tools, inappropriate content access tools, and malicious behavior access tools. Attacked devices include application-based and host-based devices.

[0051] During the testing process, real data such as network traffic and host logs generated during the testing process are collected, and relevant data obtained from the product under test are collected and compared and analyzed.

[0052] A multi-dimensional network security situation awareness assessment index is constructed, and an artificial intelligence big language model is used to compare and analyze the various perception results of the network security situation awareness product to be evaluated with the actual situation.

[0053] The final evaluation result is calculated using S=u(x_1,x_2,...,x_n), where x represents the score of each individual evaluation indicator, and u is the overall evaluation function.

[0054] In this preferred embodiment, the multi-dimensional network security situation awareness assessment indicators include:

[0055] Whether it can generate a visualized global network topology, whether network traffic data collection is complete, whether log data collection of hosts and security devices is complete, whether open port and vulnerability information is correct, whether threat intelligence is accurate, whether it can identify online assets, whether alarm push is accurate, and whether it can reconstruct a complete attack chain by correlating events through time, space, assets, and other dimensions.

[0056] In one embodiment of the present invention, the method further includes: after completing the construction of the test network environment and determining the network device to be attacked, simulating the user's online network activity behavior to generate real network traffic by simulating the user's operation behavior on the network application.

[0057] In this embodiment, user online network activity behavior is simulated to generate realistic network traffic by simulating user operations on network applications. Since there is definitely normal background traffic in a real network, if the target range only contains attack packets, the situational awareness platform can easily "identify the anomaly at a glance," leading to an inflated detection rate. However, by mixing in real network traffic generated by simulated user behavior, the test conditions become closer to reality, making the final measured detection rate, false alarm rate, and stress resistance more accurate.

[0058] In one embodiment of the present invention, the method further includes: starting real-time attack data streams and offline attack data streams in parallel in the test network environment; mirroring the real-time attack data streams on network devices along the attack path and forwarding them to the network security situation awareness system to be evaluated, thereby completing the collection of real-time attack traffic data; and labeling, denoising, and standardizing the offline attack data streams to generate a standardized test dataset, which is then sent to the network security situation awareness system to be evaluated.

[0059] In this embodiment of the invention, real-time attack data streams and offline attack data streams are launched in parallel in the test network environment. The evaluation system mirrors the currently occurring attack traffic (real-time attack data stream) and imports it into the network security situation awareness system under test in real time, enabling the system to perform real-time detection and test its real-time detection capabilities. On the other hand, this embodiment performs labeling, noise reduction, and normalization processing on the offline attack data stream to generate a standardized test dataset, which is then sent to the network security situation awareness system under test to measure its offline analysis capabilities.

[0060] like Figure 3 As shown, this application also provides a network security situation awareness system capability evaluation system 200, including:

[0061] The scenario determination module 201 is used to determine the evaluation purpose based on the network security situation awareness system to be evaluated, and to determine the test scenario from the preset scenario library according to the evaluation purpose;

[0062] The environment construction and management module 202 is used to construct a test network environment based on the test scenario, select simulated attack tools from a pre-built attack toolset and determine the corresponding attack tactics, select network devices to be attacked from a pre-built target toolset, and select data tools from a pre-built data toolset; wherein the attack toolset, the target toolset, and the data toolset are pre-built based on historical network security situation awareness and test data, respectively.

[0063] The test module 203 is used to generate a test process based on the test network environment, the simulated attack tool and corresponding attack tactics, the network device to be attacked and the data tool, and then execute the test process and collect test data during the test process.

[0064] The evaluation module 204 is used to construct a multi-dimensional network security situation awareness evaluation index based on the test scenario. Based on the multi-dimensional network security situation awareness evaluation index, the test data is compared and analyzed with the perception data and output perception results obtained by the network security situation awareness system to be evaluated through an artificial intelligence large language model to obtain the evaluation result.

[0065] This embodiment also includes a verification and analysis tool module, which provides a pre-built attack toolkit, the target machine toolkit, and the data toolkit.

[0066] In one embodiment, the network security situation awareness system capability evaluation system of this embodiment includes:

[0067] The verification and analysis tools module provides attack tools, target machine tools, and data tools required for testing and evaluation. During the testing phase, the system can directly call the tools in this module.

[0068] The scenario determination module is used to determine the evaluation purpose based on the network security situation awareness system to be evaluated, and to determine the test scenario from a preset scenario library according to the evaluation purpose.

[0069] The environment building and management module is used to build and manage the network environment, attacking machines and attack tactics, and target machines to be attacked required for testing. It uses container images to manage operating system images, target machine images, attack tool images, and data images.

[0070] The testing module is used to select test scenarios, invoke verification and analysis tools, automate test process orchestration, and collect data such as network traffic and host logs generated during testing.

[0071] The evaluation module is used to construct a multi-dimensional network security situation awareness evaluation index system to evaluate the comprehensive capabilities of the network security situation awareness products under test in network security attack analysis, asset risk analysis, abnormal behavior analysis, and situation display.

[0072] like Figure 4 As shown, in practical implementation, this evaluation system mainly includes the following modules:

[0073] The verification and analysis tools module provides attack tools, target machine tools, and data tools required for testing and evaluation. During the testing phase, the system can directly call these tools. This module offers three categories of verification and analysis tools: attack tools, target machine tools, and data tools. Attack tools simulate various attack scenarios to generate realistic attack characteristics. Target machine tools provide various types of target machines, including servers running various operating systems, personal computers, printers, etc. Data-based verification and analysis tools provide standardized and comprehensive data support for the testing environment through high-quality network security datasets. By deploying relevant offline attack traffic data on target machines and performing labeling, noise reduction, and normalization processing on the raw data, standardized test datasets are generated. These datasets simulate complex data scenarios in real network environments, allowing security products to asynchronously and offline analyze relevant attack data. This avoids the lack of security threat analysis capabilities in scenarios where traffic acquisition is impossible due to network environment or computing capacity limitations (such as large-scale denial-of-service attacks), and also verifies the applicability of security product detection algorithms in offline scenarios. The tool can directly transmit static data sets such as malicious traffic sets, malicious domains, inappropriate content, and sensitive data from data-driven tools to the situational awareness system to be evaluated, serving as data for its network security attack, asset risk analysis, and abnormal behavior analysis.

[0074] The environment building and management module can build and manage complex network environments, attack simulation tool environments, and target machine environments required for testing. It can quickly plan, deploy, scale, release, and schedule resources according to the requirements of attack tools, target machines, and testing scenarios. It manages operating system images, target machine images, attack tool images, and data images using container images. The operating system images support mainstream operating system categories to meet the compatibility requirements of situational awareness probe deployment, target machine deployment, and attack tool deployment.

[0075] The network environment construction and management unit primarily utilizes cloud virtualization, network topology simulation, network service simulation, and traffic simulation technologies to build the test range environment. Cloud virtualization enables efficient management and highly dynamic orchestration capabilities. Network topology simulation allows the system to create arbitrary network topologies, configure network services, and configure forwarding rules within the network range, achieving complex network topology simulation with connectivity, transparency, scalability, and practicality, supporting large-scale network experiments. Network service simulation allows the test range to simulate services such as social networks, routing protocols, service providers, domain name resolution, and public key infrastructure. The network simulator uses a discrete event model, allowing discrete steps and using events to communicate between simulated entities, thereby generating state transitions—that is, output results are generated based on event-triggered state changes. Network service simulation provides typical social media applications and application-level traffic simulation, increasing the realism of the test range scenario. In target machine environment construction, operating system images and target machine images are managed using container images. This allows for the rapid generation of various target machine types. The operating system images support mainstream operating system categories to meet the compatibility requirements of situational awareness probe deployment, target machine deployment, and attack tool deployment. Target machine images, attack tool images, and data images are configured with fixed image templates based on different testing scenarios, facilitating the rapid selection of relevant images for different scenarios and enabling fast and flexible deployment and environment construction.

[0076] The target machine environment construction unit is used to simulate actual target environments. By creating real or simulated target systems and data, it tests the effectiveness of attack tools and the capabilities of defense systems. It mainly includes three types: application-based target machines, host-based target machines, and malicious / sensitive data-based target machines. These target machines are primarily built using virtualization and containerization technologies for easy management and expansion. Among them:

[0077] Application-based target machines are designed to simulate vulnerable systems by deploying specific web applications (such as Tomcat web containers) and databases (MySQL, Redis, etc.) as targets, thus mimicking potential victims of application service systems in real-world environments.

[0078] Host-based target machines are designed to simulate real-world end-user victims by deploying specific operating systems (Linux, Windows) as the targets.

[0079] Inappropriate and sensitive data target machines are used to simulate high-value data systems by placing sensitive data within the operating system or business system. This tests whether the security system can effectively detect unauthorized data acquisition attempts on such target machines during an attack.

[0080] After the network environment and target machine are built, user behavior simulation can be achieved. Real network traffic can be generated by simulating user operations on network applications, such as simulating online social networking activities such as web browsing and microblogging, as well as sending and receiving emails.

[0081] The attack tool environment construction and management unit includes tools for quickly generating various attack machines, setting attack targets, and providing a large number of known attack behaviors and effective attack tactics specific to certain organizations for simulation. These include: vulnerability exploitation tools, malware attack tools, web application attack tools, data theft attack tools, malicious email attack tools, comprehensive orchestration attack tools, malicious address access tools, inappropriate content access tools, and malicious behavior access tools. Among these, vulnerability exploitation tools can simulate attacks targeting system and application vulnerabilities (such as buffer overflows and SQL injection), ultimately testing the vulnerability detection capabilities of security products and the timeliness and effectiveness of patch updates; malware attack tools simulate the propagation and behavior of viruses, worms, Trojans, and other malicious code, ultimately verifying the effectiveness of antivirus software, sandbox analysis tools, and malware behavior monitoring systems; and web application attack tools can simulate common attacks on websites and web services, such as XSS and CSRF. The test includes several methods, including path traversal, to ultimately verify the security product's ability to detect web application attack traffic. Data theft attack tools simulate attacks that steal sensitive data (such as passwords and confidential files), ultimately verifying the security product's ability to detect attacks related to data encryption, access control, and data leakage. Malicious email attack tools simulate phishing and spam email attacks, ultimately verifying the security product's ability to detect potential malicious attack locations such as email addresses, email body content, related web hyperlinks, and email attachments. Malicious address access tools simulate active access to malicious IPs, domains, and websites, assessing whether the security product can utilize its own threat intelligence capabilities to detect corresponding malicious behavior, thus verifying the real-time and comprehensiveness of its threat intelligence capabilities. Inappropriate content access tools simulate active access to inappropriate or sensitive content, ultimately verifying whether the security product can detect user access to various inappropriate content. Malicious behavior access tools simulate abnormal malicious login operations such as mass brute-force SSH logins and MySQL logins, ultimately verifying whether the security product can effectively detect abnormal behavior of network devices.

[0082] The testing module can construct a complete test network environment by calling the attack tools, target machine tools, and data tools of the verification and analysis tool module according to the test scenario. After deploying the network security situation awareness platform, the system automatically orchestrates the test process based on the scenario and comprehensively collects, stores, and analyzes the network traffic, host logs, and other data generated during the test.

[0083] The evaluation module constructs a multi-dimensional network security situational awareness capability evaluation index system, mainly including dimensions such as whether it can generate a visualized global network topology, whether network traffic data collection is complete, whether log data collection from hosts and security devices is complete, whether open port and vulnerability information is correct, whether threat intelligence is accurate, whether it can identify online assets, whether alarm pushes are accurate, and whether it can reconstruct a complete attack chain by correlating events through time, space, assets, and other dimensions. An artificial intelligence large language model is used to compare the perception capabilities of each evaluation dimension with actual test results to obtain a score for each item. The final evaluation result is calculated using S=u(x_1,x_2,...,x_n), where x is the score of a single evaluation index, and u is the overall evaluation function.

[0084] In specific implementation, the evaluation process of this invention is as follows:

[0085] The testers first select a test scenario based on the test objective. The system then automatically deploys the attack cluster and target machine cluster, selects data-driven tools, and automatically orchestrates the test process. It ensures proper isolation based on the test scenario. When an attack occurs, it guarantees that during the process of attack traffic traveling from the attack tool cluster to the target machine cluster, traffic mirroring can be easily performed on network devices along the attack path and forwarded to the situational awareness platform, thus completing the real-time collection of traffic data.

[0086] Testers can perform manual configuration. After automated deployment, testers can manually adjust settings, including network structure, attack cluster and target machine cluster size, and target machine type (host-type, server-type), providing more than six types of attack tools.

[0087] The system includes hundreds of known attack techniques and strategies from various organizations, which testers can configure accordingly.

[0088] Testers began implementing the tests. The system built attack tool services one by one. During this process, attack tool images needed to be selected from the image repository, and the node where the service container group resided needed to be chosen. Target machine services were then built one by one. This process also required selecting target machine images from the image repository and choosing the node where the service container group resided. Finally, data-driven tools were invoked to directly transmit static data sets, including malicious traffic sets, malicious domains, inappropriate content, and sensitive data, to the network security situation awareness system under test.

[0089] Recording and management of test process data. The system will record all attack behaviors and related data such as traffic, host logs, security devices, and ports generated during the test.

[0090] Monitoring and acquisition of network security situation awareness platform data. The system records all objective data collected and perceived by the tested platform, as well as all results derived by the platform based on this objective data, such as alarm information, asset risks, abnormal behavior analysis, and attack chain reconstruction.

[0091] Test Result Analysis. The system uses artificial intelligence to compare and analyze the actual test data and behaviors with the data and results obtained by the network security situation awareness platform. Based on the multi-dimensional network security situation awareness capability evaluation index system and overall evaluation function, the final evaluation result is obtained.

[0092] In one embodiment, a computer device is provided, such as Figure 5 As shown, it includes a memory 301, a processor 302, and a computer program stored on the memory 301 and executable on the processor 302. When the processor 302 executes the computer program, it implements the above-mentioned network security situation awareness system capability evaluation method.

[0093] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0094] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that executes the above-described network security situation awareness system capability assessment method.

[0095] Specifically, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0096] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cyber security situation awareness system capability assessment method, characterized in that, The method comprises the following steps: Based on the network security situation awareness system to be evaluated, determine the evaluation purpose, and determine the test scene from the preset scene library according to the evaluation purpose; Based on the test scene, build a test network environment, select a simulated attack tool from a pre-constructed attack tool set and determine the corresponding attack tactics, select a network device to be attacked from a pre-constructed target machine type tool set, and select a data type tool from a pre-constructed data type tool set; wherein the attack tool set, the target machine type tool set and the data type tool set are pre-constructed based on historical network security situation awareness and test data; According to the test network environment, the simulated attack tool and the corresponding attack tactics, the network device to be attacked and the data type tool, generate a test flow and execute a test process, and collect test data during the test process; Construct a multi-dimensional network security situation awareness evaluation index based on the test scene, compare and analyze the test data, the perception data obtained by the network security situation awareness system to be evaluated and the output perception result through an artificial intelligence large language model based on the multi-dimensional network security situation awareness evaluation index, and obtain an evaluation result; Wherein, the attack tool set includes vulnerability exploitation attack tools, malicious code attack tools, WEB application attack tools, data theft attack tools, malicious email attack tools, comprehensive attack tools, malicious address access tools, inappropriate content access tools, and malicious behavior access tools; The target machine type tool set is used to provide multiple types of target machines, including servers, personal computers and printers running multiple operating systems; The data type tool set is used to provide standardized and comprehensive data support for the test environment, including a malicious traffic set, a malicious domain name data set, an inappropriate content data set and a sensitive data set.

2. The cyber security situation awareness system capability assessment method of claim 1, wherein, Further comprising: Containerized deployment of operating system images, target machine type tool images, attack type tool images and data type tool images, and management of the operating system images, target machine type tool images, attack type tool images and data type tool images in the form of container images.

3. The cyber security situation awareness system capability assessment method of claim 1, wherein, Further comprising: According to the comparison result of the test data and the perception data obtained by the network security situation awareness system to be evaluated, calculate the evaluation index score of each network security situation awareness evaluation index, According to the influence degree of each network security situation awareness evaluation index on network security, design the weight of each network security situation awareness evaluation index, and calculate the evaluation result according to the evaluation index score of each network security situation awareness evaluation index and the corresponding weight.

4. The cyber security situation awareness system capability assessment method of claim 1, wherein, The multi-dimensional network security situation awareness evaluation index includes: visual global network topology generation capability, network traffic data collection integrity, log data collection integrity, open port and vulnerability information accuracy, threat intelligence accuracy, online asset identification capability, alarm pushing accuracy, and attack link restoration capability through multi-dimensional correlation events.

5. The cyber security situation awareness system capability assessment method of claim 1, wherein, Further comprising: After the test network environment is constructed and the network device to be attacked is determined, the user online network activity behavior simulation is performed to generate real network traffic through simulating the user's operation behavior on the network application.

6. The cyber security situation awareness system capability assessment method of claim 1, wherein, Also includes: The real-time attack data stream and the offline attack data stream are started in parallel in the test network environment, the traffic mirror of the real-time attack data stream is performed on the network device of the attack path and is forwarded to the network security situation awareness system to be evaluated, and the collection of the real-time attack traffic data is completed; The offline attack data stream is labeled, denoised and normalized to generate a standardized test data set, which is then sent to the network security situation awareness system to be evaluated.

7. A cyber security situation awareness system capability assessment system, characterized by, Includes: The scenario determination module is configured to determine the evaluation purpose based on the network security situation awareness system to be evaluated, and determine the test scenario from the preset scenario library according to the evaluation purpose; The environment construction and management module is configured to construct the test network environment based on the test scenario, select the simulation attack tool from the pre-constructed attack tool set and determine the corresponding attack tactics, select the network device to be attacked from the pre-constructed target machine tool set, and select the data type tool from the pre-constructed data type tool set; wherein the attack tool set, the target machine tool set and the data type tool set are pre-constructed based on historical network security situation awareness and test data; The test module is configured to generate a test process according to the test network environment, the simulation attack tool and the corresponding attack tactics, the network device to be attacked and the data type tool, execute the test process, and collect the test data in the test process; The evaluation module is configured to construct a multi-dimensional network security situation awareness evaluation index based on the test scenario, compare and analyze the test data, the awareness data obtained by the network security situation awareness system to be evaluated and the output awareness result through the artificial intelligence large language model based on the multi-dimensional network security situation awareness evaluation index, and obtain the evaluation result. The attack tool set includes vulnerability exploitation attack tools, malicious code attack tools, WEB application attack tools, data theft attack tools, malicious email attack tools, comprehensive attack tools, malicious address access tools, inappropriate content access tools, and malicious behavior access tools. The target machine tool set is configured to provide multiple types of target machines, including servers, personal computers and printers running multiple operating systems. The data type tool set is configured to provide standardized and comprehensive data support for the test environment, including malicious traffic sets, malicious domain name data sets, inappropriate content data sets and sensitive data sets.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the network security situation awareness system capability evaluation method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program for executing the network security situation awareness system capability evaluation method of any one of claims 1 to 6.

Images