Index management method and system for network security situation analysis

Abstract

The invention discloses an index management method and system for network security situation analysis, and relates to the technical field of data processing, and the method comprises the following steps: carrying out the structural analysis of original index data, storing the data into a data lake, and carrying out the primary marking of the data lake according to the data collection scale; performing secondary marking on the data lake according to the data behavior characteristics; obtaining an index data label according to the primary mark and the secondary mark, and configuring calculation logic for the index data label; the program adapter generates an index processing program corresponding to the index data label according to the calculation logic; and the scheduling engine executes task verification according to the index triggering instruction in combination with the index processing program and generates a verification log. According to the scheme, unified management and calculation scheduling of the index data are realized, the problem of inconsistent flow batch processing results is solved, and the accuracy and reliability of network security situation analysis are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to an indicator management method and system for network security situation analysis. Background Technology

[0002] Cybersecurity posture analysis metrics are key quantitative indicators for measuring cybersecurity status, crucial for monitoring security incidents and assessing risks. Their data sources are divided into real-time (e.g., Syslog logs pushed by network security devices) and offline (e.g., data from security operations centers), characterized by large data volumes, high real-time requirements, and significant diversity. In existing technologies, real-time metrics rely on stream processing engines for calculation, while offline metrics rely on batch processing engines for calculation triggered by events or timed events. Due to differences in metric management models and computational data flows, inconsistencies often arise in the calculation results of metrics with the same definitions. Stream processing computes by time-slicing, while batch processing uses SQL syntax to schedule tasks. This separate processing approach leads to poor metric maintainability, making it difficult to meet the data consistency requirements of cybersecurity posture analysis, and becoming a significant technical bottleneck in industry applications.

[0003] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to address the problem in existing network security situation analysis where real-time and offline indicators, due to the use of different computing engines and management modes, result in inconsistent calculation results for the same defined indicators. This invention proposes an indicator management method and system for network security situation analysis. The method involves: structuring and parsing the raw indicator data and storing it in a data lake; accurately characterizing data features according to collection scale and behavioral characteristics; generating indicator data labels based on tags and configuring calculation logic; generating corresponding indicator processing programs through a program adapter; and finally, using a scheduling engine combined with trigger instructions to execute task verification and generate verification logs. This achieves unified management and computation scheduling of indicator data, overcoming the inconsistency problem between stream and batch processing results, and improving the accuracy and reliability of network security situation analysis.

[0005] In a first aspect, one technical solution provided in this embodiment of the invention is: a method for managing indicators for network security situation analysis, comprising the following steps: The raw indicator data is structured and parsed and stored in a data lake. The data lake is labeled once according to the data collection scale and then labeled a second time according to the data behavior characteristics. Obtain indicator data labels based on primary and secondary labels, and configure calculation logic for indicator data labels; The program adapter generates an indicator processing program that corresponds to the indicator data labels based on the calculation logic; The scheduling engine executes task verification and generates verification logs based on the indicator trigger command and indicator processing program.

[0006] Preferably, the step of performing structured parsing on the original indicator data and storing it in the data lake includes the following steps: Real-time indicator data pushed by network security devices is obtained through API interfaces and parsed into structured data using regular expressions; the structured real-time indicator data is then cleaned and merged, and stored in a data lake using Iceberg tables. Simultaneously, offline metric data is periodically extracted, formatted and aligned using Iceberg tables, and then stored in the data lake.

[0007] Preferably, the step of labeling the data lake according to the data collection scale includes the following steps: A time scale is obtained based on the collection period of offline indicator data. Based on the resource configuration information of offline and real-time indicator data obtained within the time scale, the data lake is partitioned into storage sub-units. The data within the storage sub-units is then marked once based on the time scale.

[0008] Preferably, the secondary labeling of the data lake based on data behavior characteristics includes the following steps: Based on a pre-defined behavioral feature library, behavioral features are identified in offline and real-time indicator data in the data lake. These behavioral features include, but are not limited to, attack behavior features, asset change features, and data flow features. The offline and real-time indicator data in the data lake are clustered according to the unsupervised learning algorithm, and a corresponding behavior category label is assigned to each behavior cluster. The behavior category labels are then associated with the corresponding storage sub-units in the data lake.

[0009] Preferably, the step of obtaining indicator data labels based on primary and secondary labels and configuring calculation logic for indicator data labels includes the following steps: By performing primary and secondary labeling on the storage sub-units in the data lake, indicator data labels belonging to each group of offline or real-time indicator data are generated. The calculation logic is matched with the preset calculation logic template library based on the composite attributes of the indicator data labels. The calculation logic is bound to the corresponding indicator data tags to generate a calculation logic configuration file that includes at least source parameters, calculation rules, output parameters and calculation trigger conditions, and the calculation logic configuration file is stored in the calculation model library.

[0010] Preferably, the program adapter generates an indicator processing program with corresponding indicator data labels based on the calculation logic; including the following steps: The program adapter calls a preset syntax parser to perform structured parsing of the computation logic configuration file to obtain computation logic parameters; among them, the computation logic parameters include input source fields, output fields, standardized SQL expressions, trigger condition parameters, and target Iceberg table mapping rules; Read the composite attributes in the indicator data label to determine the data processing type corresponding to the indicator data label; call the corresponding basic program template from the preset program template library according to the data processing type; dynamically inject the parsed calculation logic parameters into the basic program template; The program adapter calls the syntax validator to perform syntax and consistency checks on the base program template after the injection parameters. Based on the base program template that passes the checks, it generates an executable index processing program and stores it in the program repository.

[0011] Preferably, the corresponding basic program template is called from the preset program template library according to the data processing type, including the following steps: If the data processing type is real-time indicator data, call the Flink stream processing program template as the base program template; If the data processing type is offline indicator data, use the Spark batch processing program template as the base program template.

[0012] Preferably, the scheduling engine performs task verification and generates verification logs based on the indicator triggering command and the indicator processing program, including the following steps: The scheduling engine receives metric trigger commands and extracts the command trigger type, metric identifier, and trigger timestamp from the command through the command parser. Based on the indicator identifier, match the corresponding indicator processing program in the program repository to obtain the associated computing engine type, resource configuration information, and data dependency information of the program; Based on the preset verification rule base, the program performs compatibility verification on the computing engine type associated with the program, adaptability verification on resource configuration information, and integrity verification on data dependency information. If the consistency verification passes, the scheduling engine generates a task execution certificate and pushes the indicator processing program to the corresponding task execution node according to the computing engine type, while recording the task start time and node allocation information. If any check item fails, the scheduling engine will handle it according to the preset exception handling strategy; The entire verification process is recorded in real time and a structured verification log is generated. The verification log is then partitioned by time and stored in the Iceberg log table, and an associated index is established with the execution status of the corresponding task.

[0013] Preferably, the exception handling strategy includes: for recoverable exceptions, repeating the verification according to a set number of retries; for unrecoverable exceptions, triggering an alarm notification and terminating the task startup process.

[0014] Secondly, an embodiment of the present invention also provides a technical solution: an indicator management system, applicable to the aforementioned indicator management method for network security situation analysis, comprising: Data acquisition module: performs structured parsing on raw indicator data and stores it in the data lake; Data labeling module: performs primary labeling of the data lake based on the data collection scale; performs secondary labeling of the data lake based on data behavior characteristics; Data configuration module: Obtain indicator data labels based on primary and secondary labels, and configure calculation logic for indicator data labels; Program generation module: An indicator processing program that generates corresponding indicator data labels based on the calculation logic; Execution module: Performs task verification and generates verification logs based on indicator trigger instructions and indicator processing programs.

[0015] The beneficial effects of this invention are: (1) To address the issue of inconsistent results for the same defined indicators in existing network security situation analysis due to the separation of computing engines and management models between real-time and offline indicators, this application proposes a unified indicator management and computing scheduling scheme. The scheme involves structured parsing of the original indicator data and unified storage in a data lake using Iceberg tables. A primary labeling is performed based on the data collection scale (time scale + resource segmentation), followed by a secondary labeling based on a behavioral feature library and unsupervised learning clustering. This results in indicator data labels that accurately characterize the spatiotemporal and behavioral attributes of the data. A unified computing logic is then configured for each label, and a program adapter dynamically injects Flink / Spark templates to generate an adapted program based on the data type. Finally, the scheduling engine performs unified verification and execution. In this process, the secondary labeling standardizes the description of data features, the adaptability design of the computing logic and program generation ensures consistency in batch processing rules, and the unified verification by the scheduling engine guarantees unified execution standards. These three elements work together to eliminate sources of difference across the entire chain from data storage and feature description to computing execution, achieving consistency in the calculation results of indicators with the same definition. (2) To address the issue of poor maintainability caused by the fragmented management of indicators in existing technologies, this application proposes a standardized management scheme for the entire process. This scheme constructs a closed-loop management mechanism of "data tagging - logical configuration - program generation - scheduling execution": primary and secondary tagging form a structured tagging system, making data characteristics clearly traceable; the computational logic template library and program template library achieve modular configuration and generation, reducing redundant development; and the scheduling engine's verification logs and associated indexes enable observability of task execution status. These technical features, dynamically linked through standardized templates and automated processes, replace the traditional fragmented management model of stream-batch separation, significantly reducing the maintenance costs of indicator definition, calculation rules, and execution processes, and improving the maintainability and observability of the network security indicator system.

[0016] (3) To address the problem of insufficient calculation accuracy caused by the large volume and high diversity of network security indicator data, this application proposes a precise data feature characterization and adaptive calculation scheme. First, a primary labeling method is used to segment storage sub-units based on the collection cycle and resource configuration, ensuring the orderly division of data in terms of time and resources. Secondary labeling combines a pre-set behavioral feature library with unsupervised learning to cluster and identify features such as attack behavior and asset changes, achieving accurate classification of data behavioral attributes. The composite label generated by both methods provides a precise basis for the configuration of the calculation logic, enabling the program adapter to generate programs adapted to real-time streaming or offline batch processing. In this dynamic process, the multi-dimensional precise characterization of data features lays the foundation for the adaptive configuration of the calculation logic, effectively improving the accuracy of indicator calculation in complex network environments and enhancing the reliability of network security situation analysis.

[0017] The above description of the invention is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0018] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0019] Figure 1 This is a flowchart of an indicator management method for network security situation analysis according to the present invention.

[0020] Figure 2 This is a block diagram of an indicator management system according to the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only one preferred embodiment of this invention and are only used to explain this invention. They do not limit the scope of protection of this invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0022] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures; the process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0023] Example 1: As Figure 1 As shown, a method for managing indicators for network security situation analysis includes the following steps: The raw indicator data is structured and parsed and stored in a data lake. The data lake is labeled once according to the data collection scale and then labeled a second time according to the data behavior characteristics.

[0024] As an optional embodiment, the step of performing structured parsing on the original indicator data and storing it in the data lake includes the following steps: Real-time indicator data pushed by network security devices is obtained through API interfaces and parsed into structured data using regular expressions; the structured real-time indicator data is then cleaned and merged, and stored in a data lake using Iceberg tables. Simultaneously, offline metric data is periodically extracted, formatted and aligned using Iceberg tables, and then stored in the data lake.

[0025] Understandably, this embodiment employs a technique of obtaining real-time indicator data through an API interface, parsing it into structured data using regular expressions, cleaning and merging it, and storing it in an Iceberg table in the data lake. Simultaneously, offline indicator data is periodically extracted, mapped and formatted, and also stored in an Iceberg table in the data lake. This achieves unified structured management of real-time and offline indicator data at the storage layer. Regular expression parsing ensures efficient structured transformation of real-time data, field mapping and format alignment guarantee structural consistency between offline and real-time data, and the unified storage format of the Iceberg table lays the foundation for subsequent labeling and calculation based on the same data structure. This solves the compatibility issues in subsequent processing caused by differences in storage formats between traditional batch and stream data, improving the effectiveness and standardization of data integration.

[0026] As an optional embodiment, the step of labeling the data lake according to the data collection scale includes the following steps: A time scale is obtained based on the collection period of offline indicator data. Based on the resource configuration information of offline and real-time indicator data obtained within the time scale, the data lake is partitioned into storage sub-units. The data within the storage sub-units is then marked once based on the time scale.

[0027] It is understandable that this embodiment uses the collection period of offline indicator data as a time scale, combines the resource configuration information of real-time and offline indicator data within this time scale to perform resource partitioning of the data lake to divide it into storage sub-units, and marks the data in each storage sub-unit according to the time scale. This technical means realizes the standardized division of the spatiotemporal dimensions of data lake storage. Among them, the time scale provides a unified time reference for real-time and offline data, resource partitioning based on resource configuration information ensures the reasonable division of storage sub-units, and the marking of storage sub-units by the time scale realizes the accurate association between data and time attributes. This solves the problem of disordered indicator data timing caused by inconsistent time dimensions in traditional data storage, and provides a structured data foundation for subsequent indicator calculation and analysis based on the time dimension.

[0028] As an optional embodiment, the secondary labeling of the data lake based on data behavior characteristics includes the following steps: Based on a pre-defined behavioral feature library, behavioral features are identified in offline and real-time indicator data in the data lake. These behavioral features include, but are not limited to, attack behavior features, asset change features, and data flow features. The offline and real-time indicator data in the data lake are clustered according to the unsupervised learning algorithm, and a corresponding behavior category label is assigned to each behavior cluster. The behavior category labels are then associated with the corresponding storage sub-units in the data lake.

[0029] Understandably, this embodiment employs a pre-defined behavioral feature library to identify behavioral characteristics such as attack behavior, asset changes, and data flow in real-time and offline indicator data within the data lake. It uses an unsupervised learning algorithm to cluster the data by behavior and assign category labels to each cluster. Then, it performs a secondary association labeling of the behavioral category labels with corresponding storage sub-units. This technique achieves precise characterization and structured labeling of data behavioral attributes. The pre-defined behavioral feature library provides a benchmark for feature identification, the unsupervised learning algorithm automatically mines hidden behavioral patterns to achieve efficient clustering, and the association between labels and storage sub-units binds behavioral features to storage units in the spatiotemporal dimension. This ensures that the data not only contains time and resource attributes but also possesses clear behavioral feature identifiers, solving the problem of ambiguous behavioral feature descriptions in traditional data management and providing fine-grained feature basis for the precise configuration of subsequent computational logic.

[0030] For example, a behavioral feature library must first be pre-built, containing attack behavior characteristics (such as abnormal port scanning and malicious code transmission signatures), asset change characteristics (such as server configuration modification log formats and terminal device access / disconnection identifiers), and data flow characteristics (such as cross-network segment data transmission rules and sensitive data encryption / decryption behavior patterns). When the data lake receives structured, parsed real-time indicator data (such as abnormal connection logs pushed by the firewall) and offline indicator data (such as asset change records exported by the security operations center), the system calls a feature matching algorithm to compare the two types of data one by one based on the behavioral feature library, identifying those that match "abnormal port scanning" or "server configuration modification." The system first obtains raw data with features such as [specific characteristics]. Then, an unsupervised learning algorithm (such as K-means clustering) automatically groups the data with identified features, clustering abnormal connection logs with similar attack patterns into "attack behavior clusters" and asset change records into "asset change behavior clusters," and assigning behavioral category labels such as "attack-port scanning" and "asset-configuration change" to each cluster. Finally, the system associates and binds these labels with the storage sub-units in the data lake that have been labeled and divided in the first step (such as resource sub-units under the time scale "2024-07-29 08:00-09:00") to complete the second labeling, so that the storage sub-units simultaneously carry time, resource, and behavioral feature attributes.

[0031] Obtain indicator data labels based on primary and secondary labels, and configure calculation logic for indicator data labels.

[0032] As an optional embodiment, the step of obtaining indicator data labels based on primary and secondary labels and configuring calculation logic for the indicator data labels includes the following steps: By performing primary and secondary labeling on the storage sub-units in the data lake, indicator data labels belonging to each group of offline or real-time indicator data are generated. The calculation logic is matched with the preset calculation logic template library based on the composite attributes of the indicator data labels. The calculation logic is bound to the corresponding indicator data tags to generate a calculation logic configuration file that includes at least source parameters, calculation rules, output parameters and calculation trigger conditions, and the calculation logic configuration file is stored in the calculation model library.

[0033] It is understood that this embodiment employs a technique that combines primary tagging (time scale and resource configuration information) and secondary tagging (behavioral feature tags) to generate composite indicator data tags covering spatiotemporal attributes, resource attributes, and behavioral characteristics. Based on the composite attributes of the tags, corresponding computational logic is matched from a preset computational logic template library. Then, the computational logic is bound to the tags to generate a configuration file containing source parameters, computational rules, output parameters, and triggering conditions, which is then stored in the computational model library. This technique achieves precise adaptation and standardized management of computational logic and data features. Among these techniques, the composite tags provide multi-dimensional feature basis for computational logic matching, ensuring consistency between logic and data attributes. The preset template library ensures the standardization of computational rules, and the generation and storage of configuration files enable traceability and reuse of computational logic. This solves the problems of disconnect between computational logic and data features and chaotic rules in traditional batch processing, laying a unified logical foundation for the automatic generation of subsequent indicator processing programs.

[0034] For example, firstly, the system extracts information about storage sub-units in the data lake that have undergone primary and secondary labeling. Assume a storage sub-unit is labeled as "2024-07-29 10:00-11:00_Firewall Resource Pool A" (including time scale and resource configuration attributes) at first, and "Attack Behavior - Abnormal IP Connection" at second, (including behavioral characteristic attributes). The system integrates these two types of labels to generate a composite label belonging to the real-time indicator data of this sub-unit: "2024-07-29 10:00-11:00_Firewall Resource Pool A_Attack Behavior - Abnormal IP Connection". Subsequently, the system analyzes the composite attributes of this label: a time granularity of 1 hour, associated resource of firewall, and behavior type of attack-related abnormal connection. Based on this, it matches the corresponding calculation logic in a preset calculation logic template library: for example, "Abnormal IP Connection Frequency Statistics Logic Based on Time Window" (applicable to real-time risk assessment of attack behavior).

[0035] Finally, the system binds the calculation logic with the aforementioned indicator data tags to generate a calculation logic configuration file. The source parameter is set to "the original connection log of firewall resource pool A from 10:00 to 11:00 on 2024-07-29", the calculation rule is defined as "counting the number of connections from the same IP every 5 minutes", the output parameter is "abnormal IP connection frequency and risk level", the trigger condition is "starting a risk alarm when the frequency exceeds the preset threshold", and the configuration file is stored in the calculation model library to complete the entire process.

[0036] The program adapter generates an indicator processing program with corresponding indicator data labels based on the calculation logic.

[0037] As an optional embodiment, the program adapter generates an indicator processing program corresponding to the indicator data labels based on the calculation logic; including the following steps: The program adapter calls a preset syntax parser to perform structured parsing of the computation logic configuration file to obtain computation logic parameters; among them, the computation logic parameters include input source fields, output fields, standardized SQL expressions, trigger condition parameters, and target Iceberg table mapping rules; Read the composite attributes in the indicator data label to determine the data processing type corresponding to the indicator data label; call the corresponding basic program template from the preset program template library according to the data processing type; dynamically inject the parsed calculation logic parameters into the basic program template; The program adapter calls the syntax validator to perform syntax and consistency checks on the base program template after the injection parameters. Based on the base program template that passes the checks, it generates an executable index processing program and stores it in the program repository.

[0038] Understandably, this embodiment employs a program adapter that calls a syntax parser to parse the computational logic configuration file to obtain input source fields, standardized SQL expressions, and other computational logic parameters. After reading the composite attributes of indicator data tags to determine the data processing type, it calls the corresponding basic template from the program template library and dynamically injects parameters. Then, a syntax validator performs syntax and consistency checks, ultimately generating an executable indicator processing program and storing it in the program repository. This achieves automated and standardized conversion of computational logic into an adaptable executable program. Among these measures, syntax parsing ensures the accuracy of parameter extraction, matching the processing type with the template ensures the program's adaptability to real-time / offline scenarios, dynamic injection enables personalized program configuration, and the verification process eliminates syntax errors and logical contradictions. The overall process replaces the traditional manual program writing mode, ensuring that the generation of different types of indicator processing programs conforms to a unified standard, improving the efficiency and reliability of program generation, and providing a precisely adapted execution carrier for subsequent scheduling and execution.

[0039] For example, suppose a certain indicator data label is “2024-07-29 14:00-15:00_Server Cluster B_Asset Change - Software Version Update”, and its corresponding calculation logic configuration file is stored in the calculation model library. The program adapter first calls the preset syntax parser to parse the configuration file, extracting the input source fields (such as “software version log timestamp, old version number, new version number of server B”), output fields (such as “version update frequency, update success rate”), standardized SQL expressions (such as “COUNT(new version number) OVER(PARTITIONBY server IPORDERBY timestamp)”), trigger condition parameters (such as “update statistical results once per hour”), and target Iceberg table mapping rules (such as “map to the 'software update' partition of the 'asset change statistics table'”). Next, the program adapter reads the composite attributes of the indicator data label, finds that its associated data is an offline indicator (periodic asset change records of server cluster B), thus determining that the data processing type is offline processing, and then calls the Spark batch processing basic program template from the preset program template library. Subsequently, the program adapter dynamically injects the parsed computational logic parameters into the corresponding locations of the Spark template. This includes embedding SQL expressions into the template's computational logic module and mapping input / output fields to the template's data flow interface. After parameter injection, the program adapter calls a syntax validator to validate the template: checking the compatibility of the SQL expression with Spark syntax, verifying the consistency of input fields with the source data structure, and confirming that the trigger condition parameter format meets the template requirements. Once validation passes, an executable Spark batch program is generated and stored in the "Offline Asset Change Processing" directory of the program repository for subsequent use by the scheduling engine.

[0040] As an optional embodiment, the corresponding basic program template is called from a preset program template library according to the data processing type, including the following steps: If the data processing type is real-time indicator data, call the Flink stream processing program template as the base program template; If the data processing type is offline indicator data, use the Spark batch processing program template as the base program template.

[0041] It is understandable that this embodiment employs a technique of calling corresponding basic program templates from a preset program template library based on the data processing type (real-time indicator data or offline indicator data). Real-time indicator data calls the Flink stream processing program template, while offline indicator data calls the Spark batch processing program template, achieving precise adaptation between the computing engine and the data processing scenario. Among them, the Flink template, with its low latency and high throughput characteristics, adapts to the continuous processing requirements of real-time data, while the Spark template, with its efficient batch processing capabilities, adapts to the batch computing requirements of offline data. Through a type-driven template calling mechanism, both leverage the technical advantages of stream processing and batch processing respectively within a unified framework. This ensures both the efficiency and professionalism of processing different types of data, while avoiding arbitrariness in engine selection through standardized templates, laying the foundation for consistency in subsequent program generation.

[0042] The scheduling engine executes task verification and generates verification logs based on the indicator trigger command and indicator processing program.

[0043] As an optional embodiment, the scheduling engine performs task verification and generates verification logs based on the indicator triggering instruction and the indicator processing program, including the following steps: The scheduling engine receives metric trigger commands and extracts the command trigger type, metric identifier, and trigger timestamp from the command through the command parser. Based on the indicator identifier, match the corresponding indicator processing program in the program repository to obtain the associated computing engine type, resource configuration information, and data dependency information of the program; Based on the preset verification rule base, the program performs compatibility verification on the computing engine type associated with the program, adaptability verification on resource configuration information, and integrity verification on data dependency information. If the consistency verification passes, the scheduling engine generates a task execution certificate and pushes the indicator processing program to the corresponding task execution node according to the computing engine type, while recording the task start time and node allocation information. If any check item fails, the scheduling engine will handle it according to the preset exception handling strategy; The entire verification process is recorded in real time and a structured verification log is generated. The verification log is then partitioned by time and stored in the Iceberg log table, and an associated index is established with the execution status of the corresponding task.

[0044] It is understandable that this embodiment employs a scheduling engine that receives indicator trigger commands, extracts the command trigger type, indicator identifier, and trigger timestamp through an instruction parser, matches the corresponding indicator processing program in the program repository based on the indicator identifier to obtain the computing engine type, resource configuration information, and data dependency information, and then performs compatibility checks on the computing engine type, adaptability checks on the resource configuration information, and integrity checks on the data dependency information based on a preset verification rule base. If the verification passes, an execution certificate is generated and the program is pushed to the corresponding node and the information is recorded; if it fails, it is handled according to the exception policy. At the same time, the verification process is recorded in real time, generating structured logs and storing them in the log Iceberg table by time partition, and a task status association index is established. This technical means realizes full-dimensional compliance verification before the execution of indicator processing tasks and traceable management of the execution process. Among them, multi-dimensional verification ensures the compatibility, resource adaptability, and data integrity of task execution, the exception handling strategy ensures the stability of the execution process, and the structured logs and association indexes provide accurate basis for task status tracking and problem investigation, thus improving the reliability and maintainability of network security situation analysis indicator calculation tasks.

[0045] As an optional embodiment, the exception handling strategy includes: for recoverable exceptions, repeating the verification according to a set number of retries; for unrecoverable exceptions, triggering an alarm notification and terminating the task startup process.

[0046] Understandably, this embodiment employs an exception handling strategy that involves repeatedly verifying recoverable anomalies according to a set number of retries and triggering alarm notifications and terminating the task startup process for unrecoverable anomalies. This achieves differentiated and precise handling of task execution anomalies. Among these, the retry mechanism for recoverable anomalies can effectively address occasional issues such as temporary network fluctuations and excessively high instantaneous resource consumption, improving the fault tolerance and success rate of task execution. The alarm and termination strategy for unrecoverable anomalies can avoid ineffective resource consumption. At the same time, by promptly notifying relevant personnel to quickly intervene and handle fundamental issues such as underlying configuration errors and data structure corruption, the overall stability and efficiency of the indicator processing task execution process are ensured.

[0047] Example 2, another technical solution provided in this embodiment of the invention is: an indicator management system, applicable to the aforementioned indicator management method for network security situation analysis, such as... Figure 2 As shown, it includes: Data acquisition module: performs structured parsing on raw indicator data and stores it in the data lake; Data labeling module: performs primary labeling of the data lake based on the data collection scale; performs secondary labeling of the data lake based on data behavior characteristics; Data configuration module: Obtain indicator data labels based on primary and secondary labels, and configure calculation logic for indicator data labels; Program generation module: An indicator processing program that generates corresponding indicator data labels based on the calculation logic; Execution module: Performs task verification and generates verification logs based on indicator trigger instructions and indicator processing programs.

[0048] It is understandable that this embodiment adopts a collaborative module architecture in which the data acquisition module performs structured parsing of the original indicator data and stores it in the data lake; the data labeling module performs primary and secondary labeling based on the data collection scale and behavioral characteristics respectively; the data configuration module combines the two labeling to generate indicator data labels and configure the calculation logic; the program generation module generates the corresponding processing program according to the calculation logic; and the execution module executes task verification and generates verification logs according to the indicator triggering instructions. This achieves fully automated and standardized management of network security situation analysis indicators from data access, feature labeling, logic configuration to program generation and task execution. The modules form a closed-loop processing mechanism through dynamic connection of data flow and logic flow, which not only ensures the uniformity of real-time and offline indicators in terms of data structure, feature description, calculation rules and execution standards, but also improves the scalability and maintainability of the system through modular division of labor. It effectively solves the problems of data inconsistency and process fragmentation in traditional decentralized management, and ensures the accuracy and efficiency of indicator processing.

[0049] The specific embodiments described above are preferred embodiments of the indicator management method and system for network security situation analysis of the present invention, and are not intended to limit the specific scope of the present invention. The scope of the present invention includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. A method for managing indicators for network security situation analysis, characterized in that, Includes the following steps: The raw indicator data is structured and parsed and stored in a data lake. The data lake is labeled once according to the data collection scale and then labeled a second time according to the data behavior characteristics. Obtain indicator data labels based on primary and secondary labels, and configure calculation logic for indicator data labels; The program adapter generates an indicator processing program that corresponds to the indicator data labels based on the calculation logic; The scheduling engine executes task verification and generates verification logs based on the indicator trigger command and indicator processing program.

2. The method for analyzing network security situation according to claim 1, characterized in that, The process of performing structured parsing of the original indicator data and storing it in the data lake includes the following steps: Real-time indicator data pushed by network security devices is obtained through API interfaces and parsed into structured data using regular expressions; The structured real-time indicator data is cleaned and merged, and then formatted and stored in the data lake using Iceberg tables; Simultaneously, offline metric data is periodically extracted, formatted and aligned using Iceberg tables, and then stored in the data lake.

3. The indicator management method for network security situation analysis according to claim 2, characterized in that, The step of labeling the data lake according to the data collection scale includes the following steps: A time scale is obtained based on the collection period of offline indicator data. Based on the resource configuration information of offline and real-time indicator data obtained within the time scale, the data lake is partitioned into storage sub-units. The data within the storage sub-units is then marked once based on the time scale.

4. A method for managing indicators for network security situation analysis according to claim 1 or 3, characterized in that, The secondary labeling of the data lake based on data behavior characteristics includes the following steps: Based on a pre-defined behavioral feature library, behavioral features are identified in offline and real-time indicator data in the data lake. These behavioral features include, but are not limited to, attack behavior features, asset change features, and data flow features. Based on the unsupervised learning algorithm, offline and real-time indicator data in the data lake are clustered by behavior, and a corresponding behavior category label is assigned to each behavior cluster. The behavior category labels are then associated with the corresponding storage sub-units in the data lake.

5. The indicator management method for network security situation analysis according to claim 2, characterized in that, The process of obtaining indicator data labels based on primary and secondary labels and configuring calculation logic for the indicator data labels includes the following steps: By performing primary and secondary labeling on the storage sub-units in the data lake, indicator data labels belonging to each group of offline or real-time indicator data are generated. The calculation logic is matched with the preset calculation logic template library based on the composite attributes of the indicator data labels. The calculation logic is bound to the corresponding indicator data tags to generate a calculation logic configuration file that includes at least source parameters, calculation rules, output parameters and calculation trigger conditions, and the calculation logic configuration file is stored in the calculation model library.

6. The indicator management method for network security situation analysis according to claim 1, characterized in that, The program adapter generates an indicator processing program with corresponding indicator data labels based on the calculation logic; including the following steps: The program adapter calls a preset syntax parser to perform structured parsing of the computation logic configuration file to obtain computation logic parameters; among which, the computation logic parameters include input source fields, output fields, standardized SQL expressions, trigger condition parameters, and target Iceberg table mapping rules; Read the composite attributes in the indicator data label to determine the data processing type corresponding to the indicator data label; call the corresponding basic program template from the preset program template library according to the data processing type; dynamically inject the parsed calculation logic parameters into the basic program template; The program adapter calls the syntax validator to perform syntax and consistency checks on the base program template after the injection parameters. Based on the base program template that passes the checks, it generates an executable index processing program and stores it in the program repository.

7. The indicator management method for network security situation analysis according to claim 6, characterized in that, The process involves calling the corresponding basic program template from the preset program template library based on the data processing type, including the following steps: If the data processing type is real-time metric data, call the Flink stream processing program template as the base program template; If the data processing type is offline indicator data, use the Spark batch processing program template as the base program template.

8. The indicator management method for network security situation analysis according to claim 6, characterized in that, The scheduling engine performs task verification and generates verification logs based on the indicator triggering command and indicator processing program, including the following steps: The scheduling engine receives metric trigger commands and extracts the command trigger type, metric identifier, and trigger timestamp from the command through the command parser. Based on the indicator identifier, match the corresponding indicator processing program in the program repository to obtain the associated computing engine type, resource configuration information, and data dependency information of the program; Based on the preset verification rule base, the program performs compatibility verification on the computing engine type associated with the program, adaptability verification on resource configuration information, and integrity verification on data dependency information. If the consistency verification passes, the scheduling engine generates a task execution certificate and pushes the indicator processing program to the corresponding task execution node according to the computing engine type, while recording the task start time and node allocation information. If any check item fails, the scheduling engine will handle it according to the preset exception handling strategy; The entire verification process is recorded in real time and a structured verification log is generated. The verification log is then partitioned by time and stored in the Iceberg log table, and an associated index is established with the execution status of the corresponding task.

9. A method for managing indicators for network security situation analysis according to claim 8, characterized in that, The exception handling strategy includes: for recoverable exceptions, repeating the verification according to the set number of retries; for unrecoverable exceptions, triggering an alarm notification and terminating the task startup process.

10. An indicator management system, applicable to the indicator management method for network security situation analysis as described in any one of claims 1 to 9, characterized in that, include: Data acquisition module: performs structured parsing on raw indicator data and stores it in the data lake; Data labeling module: performs primary labeling of the data lake based on the data collection scale; performs secondary labeling of the data lake based on data behavior characteristics; Data configuration module: Obtain indicator data labels based on primary and secondary labels, and configure calculation logic for indicator data labels; Program generation module: An indicator processing program that generates corresponding indicator data labels based on the calculation logic; Execution module: Performs task verification and generates verification logs based on indicator trigger instructions and indicator processing programs.

Images