Network security monitoring and early warning method based on asset analysis

Abstract

The invention relates to the technical field of data security monitoring, and provides a network security monitoring and early warning method based on asset analysis. Comprises: constructing an asset center; wherein the asset center is formed by accessing an asset logic data model to user asset data; configuring a safety monitoring network for the asset center based on Flink state monitoring, determining an execution node and a data node of the asset center, and setting an early warning analysis strategy for analysis, research and judgment according to the execution node and the data node; and analyzing the user asset data according to the early warning analysis strategy, and judging whether an early warning event exists or not. According to the invention, comprehensive perception and rapid processing of the network security situation can be realized on the network security effectiveness of asset convergence, notification and early warning, threat monitoring, co-processing, coordinated command and the like.

Description

Technical Field

[0001] The present invention relates to the field of network security monitoring, and in particular to a network security monitoring and early warning method based on asset analysis. Background Art

[0002] With the application of new technologies such as cloud computing and big data, the network scale is getting larger and larger, the number of network nodes is increasing, the data flow is large, and there are multiple heterogeneous network environments and application platforms. As network intrusions and attacks are developing towards scale and complexity, network security work has evolved from passive and emergency security protection for local information systems to security planning and layout for the entire network space with the rapid development of information technology. At present, the generally accepted network security monitoring and early warning model is the JDL (Joint Directors of Laboratories) model based on the concept of data fusion, which mainly includes the following two key technologies.

[0003] (1) Aggregation and fusion technology of massive multi-dimensional heterogeneous data At present, in large-scale networks, network security data and log data are generated by massive devices and multiple application systems, and these security data and log data lack unified standards and associations. Data analysis based on this cannot obtain global and accurate analysis results. New network security analysis and monitoring and early warning require that the analysis of network security data can break the traditional single mode, break the isolation between tables and rows, integrate the data into a whole, perform global association analysis from the whole, perform high-performance processing on the whole data, and perform multi-dimensional cutting and visualization of data in an interactive form.

[0004] (2) Multi-type network security threat assessment technology Starting from multiple data such as traffic, domain names, messages and malicious codes, it effectively processes threat data from Internet probes, terminals, cloud computing and big data platforms, decomposes abnormal behaviors hidden in different types of data, and conducts multi-level detection of security elements such as traffic, domain names, messages and malicious codes.

[0005] but: Existing network security monitoring and early warning technologies are all based on traditional data access traffic for security monitoring and early warning, and cannot effectively adapt to data identification and specific security attack detection in ecological environment scenarios. The equipment and products of the traditional security defense system are all over the data analysis of network layers 2 to 7. Among them, the 7-layer devices related to APT attacks are mainly IDS, IPS, and auditing. The IDS and IPS responsible for 7-layer detection use the classic CIDF detection model. The core idea of ​​this model is to complete the detection of attack behaviors by relying on pattern matching of the attack feature library. On the other hand, the attack methods and technologies used by APT attacks are unknown vulnerabilities (0day), unknown malicious codes and other unknown behaviors. In this case, IDS and IPS that rely on known features and known behavior patterns for detection cannot detect APT attacks in theory when they cannot predict attack features and attack behavior patterns.

[0006] The traditional approach to security incident analysis is to traverse the alarm logs of various security devices and try to find the correlation between them. However, in actual combat, especially after the attacker uses certain means to destroy evidence, traditional analysis methods and traditional security devices are usually unable to effectively detect each stage of APT attacks. Moreover, there are usually the following difficulties. On the one hand, storage is inconvenient, and the full flow data generated every day will take up too much storage space. On the other hand, the full flow data contains structured data, unstructured data, and data in multiple formats, which cannot be directly formatted and retrieved, so security personnel cannot find valuable information from the massive data.

[0007] Currently, most security monitoring and early warning products on the market can achieve different degrees of threat detection and alarms, but they are relatively weak in how to quickly and effectively handle the identified alarms. Usually, they rely on manual intervention to remember certain attack IPs or victim IPs, and log in to firewalls, antivirus systems and other platforms for manual handling. This response and handling method is complex and inefficient, and is prone to more serious security incidents due to misoperation. Summary of the invention

[0008] The present invention proposes a network security monitoring and early warning method based on asset analysis, which is used to solve the problems that data identification and specific security attack detection in ecological environment scenarios cannot be effectively adapted, storage is inconvenient, the full flow data generated every day will take up too much storage space, and the full flow data includes structured data, unstructured data and data in multiple formats.

[0009] The present invention proposes a network security monitoring and early warning method based on asset analysis, comprising: Build an asset center; the asset center is composed of an asset logic data model connected to user asset data; Based on Flink status monitoring, configure a security monitoring network for the asset center and determine the execution nodes and data nodes of the asset center; According to the execution nodes and data nodes, set up early warning analysis strategies for analysis and judgment; According to the early warning analysis strategy, analyze the user asset data to determine whether there is an early warning event.

[0010] Preferably, the building of the asset center further comprises: Pre-build the first fine-grained model of the asset overview; Through the first fine-grained model, configure the search conditions, query the asset information of any user, and obtain the asset aggregation information; According to the asset aggregation information, set up distributed asset reporting ports and asset automatic collection equipment to obtain the target asset data of any user; Construct a first network database of target asset data, perform asset data confidence calculations, and determine asset status.

[0011] Preferably, configuring a safety monitoring network for the asset center based on Flink status monitoring includes: Pre-build the Flink operating architecture and Flink programming model; According to the Flink operating architecture, a stream computing processing engine is built. The stream computing processing engine is used for stream processing and batch processing. According to the Flink programming model and streaming computing processing engine, asynchronous access to local state is stored and identified.

[0012] Preferably, the security monitoring network is configured with a log retrieval component, an alarm overview component, an alarm management component and a security event feedback component; The log retrieval component is used to perform real-time retrieval of the original log library to determine whether there is an alarm event. The real-time retrieval includes advanced mode and quick mode. In advanced mode, the query conditions entered by the user are used to retrieve alarm events. In quick mode, the user directly searches through predefined search conditions. The alarm overview component is used to count alarm events according to preset range conditions; The alarm management component is used to analyze the data source connected to the asset center and determine whether there is an alarm event; The security incident feedback component is used to perform closed-loop processing of key network security incidents through a preset key network security incident identification mechanism; wherein the closed-loop processing includes incident discovery, incident notification, incident rectification and incident feedback.

[0013] Preferably, before determining the execution node and data node of the asset center, the method further includes: Build an interactive search engine based on a distributed non-relational database; According to the interactive search engine, the asset data of the asset center is classified according to the documents; According to data classification, event data that has passed data quality assessment is determined; wherein the event data includes normalized data fields and original logs, and the event data corresponds to an index file; According to the index file, the time data is sharded to generate a horizontal table; each shard of the horizontal table corresponds to a search engine, and each shard is configured with multiple copies.

[0014] Preferably, the execution node is used to receive an index request and calculate the shard into which the event document corresponding to the index request should be loaded; the data node is used to send the index request to the corresponding target node and generate inverted index information of the event document.

[0015] Preferably, after determining the execution node and data node of the asset center, the method further includes: Establish analysis models, including multi-dimensional analysis models, attack chain analysis models, and high-risk IP analysis models; The multidimensional analysis model is used to obtain the traffic behavior of any user asset data in the data center, identify key entities, and determine each security risk event based on the target relationship network between key entities and attack events; Attack chain analysis is used to determine the attack source and victim address of the network attack. Based on the attack source and victim address, the attack process on user asset data is restored to identify high-risk events. High-risk IP analysis is used to purify the user asset data of the data center and determine whether an accident has occurred based on the purified high-quality small data.

[0016] Preferably, the early warning analysis strategy for setting analysis and judgment includes: Obtain user access information supervised by the asset center and build a network security ecosystem based on access users; Configure the plan arrangement according to the network security ecosystem; the plan arrangement includes the work plan and the network security incident emergency plan; When a behavioral event triggers a plan, the early warning analysis strategy for the corresponding behavioral event is output.

[0017] Preferably, the asset center is also used for: Pre-set custom function interface and business data for business configuration; According to the functional interface, configure the content editor to edit and publish the data in the asset center; Build a business knowledge base based on business data; the business knowledge base includes the IP address library and the ATT&CK library; Generate a data warehouse based on standardized documents based on business knowledge base, data editing and data publishing.

[0018] Preferably, the asset center is also used to build a user system; wherein the user system includes a permission system and an internal account system; Configure internal permissions for different users according to the permission system; According to the internal account system, account configuration is performed for user data sources, where account configuration includes: login factor configuration, high-risk operation factor configuration, and basic authentication factor configuration.

[0019] The beneficial effects of the present invention are: In terms of asset aggregation, network asset information is aggregated through system docking, active detection, passive identification, user reporting, etc., to establish an asset information database, and update it in a timely manner according to asset changes. The network assets of the managed units are fully understood.

[0020] In terms of notification and warning, the security incident data such as website vulnerabilities, Trojans, botnets, website tampering, website impersonation, website hijacking, dark links, APT attacks, etc. generated by network security monitoring and analysis, as well as manually imported externally acquired security incidents, will be promptly reported and warned to the relevant units involved, and the units involved will be coordinated to complete the closed-loop work of handling network security risks and network security incidents.

[0021] In threat monitoring, based on asset information and network security information, relevant technical means are used to timely discover network security incidents, identify network security risks and threats, conduct comprehensive analysis, research and dynamic tracking of network security situation from multiple dimensions, and realize the transformation of network security situation from preliminary perception to comprehensive perception.

[0022] In terms of collaborative handling, scenario-based business process management is realized for the entire work cycle, including daily event notification, important activity support, emergency response to sudden incidents, etc., as well as the collaborative scheduling and unified orchestration of various dispatching and command tasks in the process.

[0023] In terms of coordination and command, it supports issuing coordination and command instructions such as incident notification, early warning notification, emergency disposal, special event analysis, etc. to the managed units, tracking the execution of instructions, and receiving receipts and feedback on disposal results.

[0024] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0025] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 A method flow chart of a network security monitoring and early warning method based on asset analysis in an embodiment of the present invention; Figure 2 This is a diagram of the Flink operation architecture in an embodiment of the present invention; Figure 3 This is a functional display diagram of the Flink programming model in an embodiment of the present invention; Figure 4 This is a diagram of the instruction coordination service architecture in an embodiment of the present invention; Figure 5 This is a general architecture diagram of the asset model in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0028] like Figure 1 As shown, this embodiment provides a network security monitoring and early warning method based on asset analysis, including: Build an asset center; the asset center is composed of an asset logic data model connected to user asset data; Based on Flink status monitoring, configure a security monitoring network for the asset center and determine the execution nodes and data nodes of the asset center; According to the execution nodes and data nodes, set up early warning analysis strategies for analysis and judgment; According to the early warning analysis strategy, analyze the user asset data to determine whether there is an early warning event.

[0029] The principle of the above technical solution is: In the actual implementation process, the present invention is based on the network security monitoring and early warning technology in the field of ecological environment, and establishes a cyberspace security governance system integrating monitoring, early warning, disposal, emergency response, command and coordination, so as to grasp the overall network security situation in real time, timely grasp the network attack situation, grasp the relevant intelligence information, and timely report and warn of network security threats; once a major network security incident occurs, it can quickly organize research and judgment, and quickly command and dispatch relevant personnel to carry out emergency disposal.

[0030] The beneficial effects of the above technical solution are: In terms of asset aggregation, network asset information is aggregated through system docking, active detection, passive identification, user reporting, etc., to establish an asset information database, and update it in a timely manner according to asset changes. The network assets of the managed units are fully understood.

[0031] In terms of notification and warning, the security incident data such as website vulnerabilities, Trojans, botnets, website tampering, website impersonation, website hijacking, dark links, APT attacks, etc. generated by network security monitoring and analysis, as well as manually imported externally acquired security incidents, will be promptly reported and warned to the relevant units involved, and the units involved will be coordinated to complete the closed-loop work of handling network security risks and network security incidents.

[0032] In threat monitoring, based on asset information and network security information, relevant technical means are used to timely discover network security incidents, identify network security risks and threats, conduct comprehensive analysis, research and dynamic tracking of network security situation from multiple dimensions, and realize the transformation of network security situation from preliminary perception to comprehensive perception.

[0033] In terms of collaborative handling, scenario-based business process management is realized for the entire work cycle, including daily event notification, important activity support, emergency response to sudden incidents, etc., as well as the collaborative scheduling and unified orchestration of various dispatching and command tasks in the process.

[0034] In terms of coordination and command, it supports issuing coordination and command instructions such as incident notification, early warning notification, emergency response, special event analysis, etc. to the managed units, tracking the execution of instructions, receiving receipts and feedback on the handling results, such as Figure 4 shown.

[0035] As an embodiment of the present invention: the construction of the asset center further includes: Pre-build the first fine-grained model of the asset overview; Through the first fine-grained model, configure the search conditions, query the asset information of any user, and obtain the asset aggregation information; According to the asset aggregation information, set up distributed asset reporting ports and asset automatic collection equipment to obtain the target asset data of any user; Construct a first network database of target asset data, perform asset data confidence calculations, and determine asset status.

[0036] The principle of the above technical solution is: In the actual implementation process, as shown in the attached Figure 5As shown, the asset center corresponds to an asset model. The asset center uses a unified asset logical data model to access, clean and enrich asset data, forming high-value asset library information for customers. At the same time, it analyzes the vulnerabilities of assets through vulnerability knowledge combined with leak scanning equipment, promotes asset reinforcement, and ensures asset security and compliance through the filing of security assets, security self-inspection and evaluation information management.

[0037] It supports the aggregation of network assets, including key information infrastructure and public space assets, through active detection, traffic analysis, manual reporting, data import, etc., and integrates multi-source heterogeneous data through a unified asset data model to form a network asset library with systems, websites, computing equipment, software, services, computer rooms, and cloud platforms as the main body, establishes connections between key assets and their affiliated units, four-party units, and operators, and forms an asset knowledge map.

[0038] For core key assets, relevant operation means are provided to automatically or semi-automatically maintain asset changes. At the same time, the reporting management of key infrastructure asset information is completed, and the annual key infrastructure asset situation is uniformly analyzed. The system supports asset overview, asset retrieval, asset reporting, asset access configuration, asset management, asset discovery, asset operation, vulnerability management, vulnerability library management, topology management, and security management.

[0039] Based on NISTIR 7693, combined with high-quality asset information obtained through management means and large-scale data obtained through technical means, we complete the aggregation and fusion process of multi-source asset data and basic asset management capabilities, assisting operators to manage assets efficiently, understand their assets, and understand asset conditions and vulnerability trends.

[0040] The specific roles of the asset center include: (1) Asset Overview Conduct statistical analysis on assets, including quantitative statistics of various types of assets, fine-grained analysis of units, servers, software, and services, and analysis of asset trends, assets to be operated, asset operation status, entity dimensions, and asset operation processes in IP type dimensions.

[0041] (2) Asset retrieval Search and query all managed assets based on different search conditions. It supports two query modes: quick mode and advanced mode. It also supports saving search conditions. The search results can be classified and displayed according to the asset type. It also supports aggregated statistical display based on the asset fields.

[0042] (3) Asset reporting Through distributed asset reporting, asset information review and management processes, asset information can be entered into the database, online changes of asset information can be supported, and the freshness and quality of assets can be guaranteed.

[0043] (4) Asset access configuration By configuring asset collection equipment, you can realize automatic collection of assets from multiple units and network segments. Support DHCP collection policy configuration, and uniquely identify the collected assets to distinguish between assets identified by asset discovery tools and assets synchronized by third-party systems.

[0044] (5) Asset management Assets are managed and maintained according to different types of entities, and management of units, equipment, software, personnel, systems, websites, computer rooms, cloud platforms, networks, and database assets is achieved, including operations such as adding, editing, deleting, importing, and establishing associations. Take units as an example: unit information is displayed in tree view and list view, and the number of systems, equipment, and websites of each unit is counted. The unit details show the basic information, network resources, associated assets, vulnerability information, and change records of the unit. It supports the maintenance of basic unit information, responsible persons, contacts, leaders in charge of network security, leaders of network security responsible departments, contacts of network security responsible departments, network resources, and other information.

[0045] (6) Asset Discovery It realizes the management of asset storage records, confidence rule configuration and scanning tasks. It uses quick search and advanced search modes to search for original assets, configures confidence rules for asset discovery, and realizes the storage of discovered assets by configuring scanning tasks.

[0046] (7) Asset Operation Unit management is provided for unowned asset entities or asset entities with ownership conflicts during operation; three types of asset ownership rule settings are supported: ownership rules, operation rules, and ignore rules. Operations such as viewing and unit setting are supported for unowned equipment, system asset entities, and equipment asset entities with ownership conflicts during operation.

[0047] (8) Vulnerability management By associating the managed asset data with the vulnerability scan data, asset vulnerability data is generated to manage the asset vulnerability status. Supports statistical analysis of asset host vulnerabilities, web vulnerabilities, weak passwords, and configuration verification. By associating the managed asset data with the vulnerability knowledge base, asset potential vulnerability data is generated to manage the potential vulnerability status of assets.

[0048] As an embodiment of the present invention: configuring a security monitoring network for an asset center based on Flink status monitoring includes: Pre-build the Flink operating architecture and Flink programming model; According to the Flink operating architecture, a stream computing processing engine is built. The stream computing processing engine is used for stream processing and batch processing. According to the Flink programming model and streaming computing processing engine, asynchronous access to local state is stored and identified.

[0049] The principle of the above technical solution is: In the actual implementation process, Figure 2 and Figure 3 As shown in the figure, security monitoring mainly provides functions such as log retrieval, alarm preview, alarm management, security events and security risks, which can meet the needs of security personnel or business personnel to analyze and judge logs and alarms, and carry out risk warning and event notification based on alarm information. The alarm merging technology based on Flink state is adopted: As the third-generation stream computing processing engine, Apache Flink is a distributed processing framework and distributed processing engine that supports high throughput, low latency, and high performance. It is used for stateful computing on unbounded and bounded data streams. Flink can run in all common cluster environments and can perform calculations at memory speed and any scale. Currently, Flink plays a huge role in the fields of real-time computing, system monitoring, public opinion monitoring, traffic forecasting, national power grid, disease prediction, banking / financial risk control, etc.

[0050] Flink's implementation of stream processing and batch processing is completely different from some traditional solutions. It looks at stream processing and batch processing from another perspective and unifies the two: Flink fully supports stream processing, which means that the input data stream is unbounded when it is viewed as stream processing; batch processing is regarded as a special stream processing, but its input data stream is defined as bounded.

[0051] Stateful Flink programs are optimized for local state access. The state of a task is always kept in memory, and if the state size exceeds the available memory, it is saved in a disk data structure that can be accessed efficiently. Tasks perform all computations by accessing local (usually in-memory) state, resulting in very low processing latency. Flink ensures exactly-once state consistency in failure scenarios by periodically and asynchronously persisting local state.

[0052] As an embodiment of the present invention: the security monitoring network is configured with a log retrieval component, an alarm overview component, an alarm management component and a security event feedback component; The log retrieval component is used to perform real-time retrieval of the original log library to determine whether there is an alarm event. The real-time retrieval includes advanced mode and quick mode. In advanced mode, the query conditions entered by the user are used to retrieve alarm events. In quick mode, the user directly searches through predefined search conditions. The alarm overview component is used to count alarm events according to preset range conditions; The alarm management component is used to analyze the data source connected to the asset center and determine whether there is an alarm event; The security incident feedback component is used to perform closed-loop processing of key network security incidents through a preset key network security incident identification mechanism; wherein the closed-loop processing includes incident discovery, incident notification, incident rectification and incident feedback.

[0053] The principle of the above technical solution is: In the actual implementation process, log retrieval mainly performs real-time retrieval and viewing of massive alarm data, traffic data, and terminal data in the original log library, and records the retrieval conditions and process to assist network security operators in alarm analysis. Log retrieval provides users with comprehensive management of alarm logs, alarms in the alarm resource library, event records, and traffic logs. It supports global search and viewing of search results based on time periods and data types, and can analyze the global search result information. It supports two query modes, quick mode and advanced mode. The quick mode supports quick search of predefined data fields, and the advanced mode supports analysts to enter query conditions according to syntax. It supports saving search conditions as common search conditions, saving and viewing historical search condition information, using predefined query conditions for quick retrieval, and exporting the retrieved data.

[0054] The alarm overview enables summary statistical analysis of the overall situation of alarms. The system supports statistical analysis related to alarms based on the time range, including: the total number of alarms, the number of alarms of each hazard level, the first-level classification of alarms, the second-level classification of alarms, the alarm discovery trend, the ranking of intranet asset alarms, the ranking of extranet asset alarms, the ranking of network area alarms, and year-on-year / month-on-month analysis of alarms.

[0055] Alarm management provides security personnel or business personnel with the ability to manage alarms generated by access platform data sources, supports security personnel or business personnel to further analyze and judge the merged alarms, realizes real-time monitoring of network security, promptly discovers the attack activities, means and targets of international hostile forces, hacker organizations and other criminals, comprehensively monitors the information systems and networks of key security units, realizes the discovery and identification of security vulnerabilities, potential threats, and advanced threat attacks, and provides strong data support for business systems such as risk warnings and incident notifications.

[0056] The security event module provides security personnel or business personnel with system-generated events or manually entered events for management operations. It identifies key network security events and determines the subsequent business flow based on the research and judgment conclusions, and initiates the corresponding notification and disposal actions. After receiving the notification, the key information infrastructure operating unit will make timely rectifications and provide feedback on the platform, thus forming a closed loop of threat discovery, notification, rectification, and feedback.

[0057] The security risk module means that when a major network risk occurs, the platform can promptly help users obtain risk details, issue risk warnings in a timely manner, and remind various units to take prevention and control measures to avoid network security incidents.

[0058] As an embodiment of the present invention: before determining the execution node and the data node of the asset center, the method further includes: Build an interactive search engine based on a distributed non-relational database; According to the interactive search engine, the asset data of the asset center is classified according to the documents; According to data classification, event data that has passed data quality assessment is determined; wherein the event data includes normalized data fields and original logs, and the event data corresponds to an index file; According to the index file, the time data is sharded to generate a horizontal table; each shard of the horizontal table corresponds to a search engine, and each shard is configured with multiple copies.

[0059] The principle of the above technical solution is: During the specific implementation process, cloud check, multi-dimensional analysis, attack chain analysis, and high-risk IP analysis functions are realized as auxiliary means for operators to analyze and judge alarms and events.

[0060] Using interactive fractional technology based on search engines: Interactive analysis is a powerful technology that provides security analysts with manual security incident analysis.

[0061] Security issues are complex, diverse and constantly evolving. Currently, there is no fully automated security analysis solution on the market to replace human work. For many unknown security issues, experienced security analysts are still required to perform manual analysis.

[0062] In the log collection and analysis system, interactive analysis provides a working interface for security analysts, provides data according to the needs of security analysts, helps analysts to programmatically calculate data, and provides rich visualization methods to help analysts better understand the data.

[0063] The core capabilities of interactive analysis are query and retrieval.

[0064] Before the era of big data, traditional systems using relational databases encountered difficulties in querying and retrieving massive amounts of events. The query efficiency was extremely low and reached the point of being unusable during interactive analysis.

[0065] The distributed non-relational database ElasticSearch is used to implement the search engine.

[0066] ElasticSearch provides a distributed, multi-user, full-text search engine that is document-oriented. Each event or piece of data is treated as a document, and JSON is used as the document serialization format.

[0067] ElasticSearch creates index files for document data according to date. For event data, the system creates an index file every day to save event data.

[0068] When searching, the event is searched for each index file according to the query conditions. ElasticSearch collects the data-managed event data transmitted by the system message system. The managed event data includes the normalized fields and the original logs. ElasticSearch performs inverted indexing on each normalized field. For the original log part, ElasticSearch automatically performs word segmentation on the original message according to the word segmentation rules, and then performs inverted indexing on each word. The whole process completes the Mapping.

[0069] The data in an index is stored in multiple shards, which is equivalent to horizontal table partitioning. A shard itself is a complete search engine. Our documents (event record sets) are stored and indexed in shards, and ElasticSearch uses shards to achieve distribution. At the same time, the same shard can generate multiple copies, and shards and copies are deployed on different nodes to achieve high availability, high reliability and performance of data. ElasticSearch supports distributed cluster deployment. When a distributed cluster is deployed, there are two different nodes: execution nodes and data nodes. The execution node is responsible for managing all changes within the cluster, such as adding and deleting indexes, or adding and deleting nodes. The execution node does not need to be involved in document-level changes and search operations. The data node is responsible for storing data and its corresponding inverted index.

[0070] As an embodiment of the present invention: the execution node is used to receive an index request and calculate the shard into which the event document corresponding to the index request should be loaded; the data node is used to send the index request to the corresponding target node and generate an inverted index information of the event document.

[0071] The principle of the above technical solution is: Execution nodes do not need to be involved in document-level changes and searches. Data nodes are responsible for storing data and its corresponding inverted index.

[0072] When the system submits a request to index a new document to the execution node, the node will calculate which shard the new document should be added to. Each node stores information about which node each shard is stored on, so the coordination node will send the request to the corresponding node. Note that this request will be sent to the primary shard, and once the primary shard completes the indexing, it will send the request to all its replica shards in parallel to ensure that each shard holds the latest data.

[0073] The query process is generally divided into two stages: query and fetch. The task of the execution node is to broadcast the query request to all relevant shards and integrate their responses into a globally sorted result set, which is returned to the client. When a node receives a search request, the first step is to broadcast the request to the shard copy of each node in the index. The query request can be processed by a primary shard or a replica shard, and the coordinating node will poll all shard copies in subsequent requests to share the load.

[0074] Each shard will build a priority queue locally. If the client requests to return a result set of N starting from the Sth in the result sorting, each node needs to generate a result set of size S+N, so the size of the priority queue is also S+N. The shard will only return a lightweight result to the coordinating node, including the ID of each document in the result set and the information required for sorting.

[0075] The execution node will aggregate the results of all shards and perform global sorting to obtain the final query sorting result. At this point, the query phase ends. The query process obtains a sorting result, marking which documents meet the search requirements. At this time, these documents still need to be obtained and returned to the client.

[0076] The execution node determines the document that actually needs to be returned and sends a get request to the shard containing the document; the shard obtains the document and returns it to the execution node; the execution node returns the result to the client.

[0077] How does ElasticSearch achieve fast indexing? The core technology is inverted index and a series of memory reading and compression technologies.

[0078] The essence of traditional relational database search is to improve query performance by reducing the number of disk seeks, and ElasticSearch also adopts the same idea. ElasticSearch directly searches for the value of the key to be queried through memory without reading the disk. However, if there are too many keys in the index, the dictionary table will also be very large, and it is not realistic to put it in memory. Therefore, there is a key index, just like the index page in the dictionary, which keys start with A and which page they are on. It can be understood that the key index is a tree. This tree does not contain all terms, but some prefixes of the terms. Through the key index, you can quickly locate a certain deviation value in the key dictionary table, and then search sequentially from this position. Therefore, the key index table does not need to store all the keys, but only the mapping relationship between some of their prefixes and the blocks of the key dictionary table. Combined with the compression technology of FST (Finite State Transducers), the key index table can be cached in memory. After finding the block location of the corresponding dictionary table from the key index table, go to the disk to find the key, which greatly reduces the number of random disk reads.

[0079] Regarding joint search, we have just introduced the single key value index. If multiple key value indexes are jointly queried, the inverted index can use the skip list data structure to quickly perform an "AND" operation or use the stored data structure to perform a bitwise "AND" operation to obtain the result.

[0080] In short, the key to ElasticSearch indexing is to move as much content as possible from the disk into the memory, reduce the number of random disk reads (while also taking advantage of the disk's sequential read characteristics), combine a variety of sophisticated compression algorithms, and use memory in a greedy and fully utilized manner to optimize search performance.

[0081] As an embodiment of the present invention: after determining the execution node and the data node of the asset center, the method further includes: Establish analysis models, including multi-dimensional analysis models, attack chain analysis models, and high-risk IP analysis models; The multidimensional analysis model is used to obtain the traffic behavior of any user asset data in the data center, identify key entities, and determine each security risk event based on the target relationship network between key entities and attack events; Attack chain analysis is used to determine the attack source and victim address of the network attack. Based on the attack source and victim address, the attack process on user asset data is restored to identify high-risk events. High-risk IP analysis is used to purify the user asset data of the data center and determine whether an accident has occurred based on the purified high-quality small data.

[0082] The principle of the above technical solution is: In the actual implementation process, multi-dimensional analysis analyzes the behavior in the traffic and combines it with open source intelligence information to dig out the attackers behind the network security incidents. Using traffic logs, security events, PCAP packages and other information, combined with open source threat intelligence (including but not limited to malicious domain names, IPs, malware, device fingerprints, attackers\organizations) and network attack and defense knowledge bases, based on the idea of ​​ontology, from the perspective of people, places, objects, things, and organizations in cyberspace, extract key entities (attack events, attack methods, IPs, domain names, devices, files, addresses, emails, phones...) and build a relationship network between entities. Through the analysis methods such as attacker homology, IP communication, virtual-real mapping, combined with control and other means, find out the real attackers hidden behind key network security incidents, locate their positions, identities, intentions, and form attacker organizations.

[0083] Attack chain analysis is based on the monitored network attack alarm information. It uses network traffic logs, network security alarms, and manual forensics information to restore the attack process based on the attack source and victim address, understand the attacker's attack path and the status of the victim assets.

[0084] High-risk IP analysis makes analysis possible in a big data environment by purifying massive security big data into usable high-quality small data. Through repeated collisions of multi-dimensional security data such as alarms, assets, and threat intelligence, it helps customers see the analysis data, threat IP, and threat characterization; thereby greatly improving the efficiency, accuracy, and depth of the analysis business, thereby reducing threat traffic, security incidents, and security accidents.

[0085] As an embodiment of the present invention: the early warning analysis strategy for setting analysis and judgment includes: Obtain user access information supervised by the asset center and build a network security ecosystem based on access users; Configure the plan arrangement according to the network security ecosystem; the plan arrangement includes the work plan and the network security incident emergency plan; When a behavioral event triggers a plan, the early warning analysis strategy for the corresponding behavioral event is output.

[0086] The principle of the above technical solution is: In the actual implementation process, the early warning analysis strategy relies on work command, which includes plan preparation, governance resource management, as well as coordinated command of important activity support and emergency response to sudden incidents.

[0087] Governance resources integrate the national network security protection resources, including units and organizations at all levels (such as the Cyberspace Administration of China, the Public Security Bureau, the Ministry of Industry and Information Technology and other security regulatory agencies, operators, domain name service agencies, cloud platform service providers and other network infrastructure providers, Internet security companies, professional institutions), personnel (experts, security support personnel, social resources), equipment (platforms, systems, tools), means (notifications, work orders, meetings and other management means as well as disconnection, IP blocking, vulnerability scanning and other technical means) and positions (operation positions, security service positions, business positions, management positions, etc.), to build a network security ecosystem.

[0088] Plan arrangement includes work plans and network security incident emergency plans, and supports the formation of plans by atomic means, so as to realize flexible customization of business processes in different scenarios. It solves the problem of rigid model and lack of flexibility when business changes in different scenarios require customization or new business functions. For example, daily incident notifications, important event support and other businesses are inseparable from the support of process plans, and all plans or process resources in plan arrangement are used. Therefore, plan arrangement is an important business basic support.

[0089] The purpose of important event security is to dispatch network security personnel during important meetings or major events, grasp the security status of key security units, systems and websites, timely report and warn of network security risks, and efficiently handle network security incidents. The platform supports the collaboration of multiple technical support units, Internet security vendors and network security experts to ensure the network security and data security of the entire process. Through all-round and all-weather command and dispatch during the security period, it reports and warns of network security threats and incidents, quickly handles major network security incidents, realizes full-factor collection, full-factor monitoring, and flat command, and improves the security assurance capabilities of major events. During the security execution process, it supports the launch of a plan based on an attack organization launching an attack on a key unit within the security scope. The platform conducts a comprehensive assessment of the degree of harm and initiates an incident notification to the unit after manual confirmation. After the disposal is completed, the unit will feedback the disposal results to the platform. The platform establishes an emergency response file for the unit for subsequent assessment.

[0090] The major incident emergency response subsystem, through programmable digital emergency plans and machine-readable threat intelligence, combined with pre-emergency drills, can provide all-round support for the early warning response process and emergency command process for major cybersecurity incidents, coordinate and coordinate various resources, define personalized monitoring indicators and views for security incidents, monitor the development of events in real time, and improve the country's emergency command capabilities during major cybersecurity incidents.

[0091] As an embodiment of the present invention: the asset center is also used for: Pre-set custom function interface and business data for business configuration; According to the functional interface, configure the content editor to edit and publish the data in the asset center; Build a business knowledge base based on business data; the business knowledge base includes the IP address library and the ATT&CK library; Generate a data warehouse based on standardized documents based on business knowledge base, data editing and data publishing.

[0092] Business configuration enables customized configuration of data and page presentation of various businesses on the platform, so as to adapt to different types of customer needs according to actual data conditions. Configuration is performed for each business function module of the platform, including: workbench configuration, asset center application configuration, log retrieval configuration, alarm configuration, event configuration, knowledge base configuration, instruction configuration, standard document configuration, tag management, data center, analysis engine management, report center, and data aggregation.

[0093] The workbench configuration supports workbench theme configuration and card management, provides a content editor, supports editing and publishing of information in a WYSIWYG manner, and also supports configuration management of the workbench's quick entry.

[0094] The asset center application configuration manages various types of assets, including custom fields, field display, and list layout configuration, to achieve display and layout customization of various asset attributes.

[0095] Log retrieval configuration supports custom configuration of log retrieval headers, field functions, and retrieval fields.

[0096] Alarm configuration supports maintenance of field grouping in the detail view of alarm management in the threat center module, configuration of alarm page template name, alarm source, alarm generation method, alarm type, and page grouping and group field layout configuration.

[0097] Event configuration provides custom field configuration for events and supports form layout configuration for security event editing operations. Event status grouping supports custom grouping configuration for event and risk status.

[0098] The knowledge base refers to a collection of characteristic knowledge data and rule methods related to a certain business field. These data provide guiding rules and support for the aggregation, integration, analysis and mining, and value enhancement of various types of business data entering the platform. Currently, it includes the IP address library and the ATT&CK library. The IP address library is the basic knowledge base of each system in the entire system. Its data is applied to each module, mainly including unit IP address information data and IP address segment information. The ATT&CK model is a model and knowledge base that reflects the attack behavior of each attack life cycle. The model describes the attacker's methods and behaviors in a unified and structured way, and is presented using a unified framework. It can describe the strategies and methods used by the attacker, and can have a more consistent process to determine the various stages of the attack. The ATT&CK matrix model is used for the platform to manage the rules of the analysis engine.

[0099] Instruction configuration supports custom business instruction types, and supports classification management of instruction types from four sub-dimensions: form creation, stage configuration, link configuration, and global configuration.

[0100] Standard document configuration mainly configures and manages event notification templates. It supports operations such as creating, editing, cloning, configuring, deleting, previewing, exporting, and saving templates, and supports turning on / off the enabled status in the list. The default "standard template" supports editing the template content in a WYSIWYG configuration mode.

[0101] Tag management supports the definition of independent tag libraries based on business modules, and tag maintenance supports the maintenance of tag grouping, tag names, and background colors.

[0102] The data center includes metadata management, data warehouse and data services. It provides a one-stop, standardized, visualized, transparent and operational big data asset full life cycle intelligent management platform, providing legal, compliant, safe and efficient technical support for the realization of raw data assetization, data asset valueization and business dataization. Metadata management will digitize security business, establish contextual information, and provide support capabilities for endogenous data applications. Support the management of business metadata, data business types, information classification coding standards, rules and quality management. Manage business metadata to determine commonly recognized vocabulary and corresponding information assets to ensure common understanding of domain knowledge; manage the mapping relationship between data business types and data physical types in different data source types to achieve unified definition of technical metadata and business metadata; manage information classification coding standards, support functions such as adding, deleting, querying and editing information classification coding standards; based on the data quality rule base, configure different quality detection rules to meet the timeliness, accuracy, completeness, consistency and effectiveness detection of data quality. The data warehouse manages the business system data in a unified manner for easy application. The detailed layer under the data warehouse extracts and loads data from various business systems and types into the target database based on its powerful data preprocessing capabilities, achieving synchronization and integration of various business data, laying the foundation for further data processing. Data services provide API data consumption methods to help users better apply data assets to realize value.

[0103] Analysis engine management is the process of extracting valuable information from a large amount of raw security data to form alarms and ultimately obtain actionable events. This is a process of reducing the amount of data and increasing the value of data. Security analysts complete the entire data processing process with the support of system tools. The joint analysis engine receives standardized log data and produces alarms. The engine input includes logs generated by local devices and various data provided by the cloud. The alarms and events output by the engine provide statistical data for data analysts to facilitate subsequent processing.

[0104] Based on the BI engine, the report center uses tools such as data sources, data sets, and interfaces to create report views and assemble them into various report analysis dashboards. The report center supports the creation and management of system data sources, analysis views, and dashboards. Data source management enables reading and management of data sources required for analysis. It supports reading relevant data table structures and table data information from the database, supports database materialized views, and updates data according to specified cycles and frequencies; supports file data source import and verification; supports reading data from interfaces; supports data preview and data search. The report center supports configuration functions and graphical operation interfaces to help users easily develop professional data reports, and guides users to quickly assemble data reports for different demand scenarios like building Lego blocks through WYSIWYG operations. The system supports the access of multi-source heterogeneous data, and has built-in multiple layout templates. Reports can be predefined in page layout templates according to different needs, and presentation reports can also be built according to different needs. By flexibly configuring various analysis reports, the data results of multiple business subsystems such as the situation center, asset center, threat center, and analysis center can be processed and output into multiple file formats such as Excel, Word, and PDF, supporting system operators to complete analysis reports on the overall security situation, related security incidents, and business work status, and supporting the push of constructed reports via email and SMS.

[0105] Data aggregation supports two types of data sources: database data sources and HTTP data sources. By creating data sets, it provides two types of data services: synchronous and asynchronous services, and supports monitoring of the execution status of data services and data set collection tasks.

[0106] The customized large screen provides a large-screen visual decision analysis component library from a macro decision-making perspective. It divides the statistical analysis components into business entity components, analysis components, and conditional components. It realizes business scenario visualization by selecting components and customizing the layout, and supports preview, copying, deletion and other operations on the scene large screen.

[0107] The cockpit customization module supports the display customization of cockpit content, including cockpit title, resolution, zoom adaptation and background image, etc. It also supports the management and setting of cockpit scenes, such as scene mode, full range, field of view, etc.

[0108] As an embodiment of the present invention: the asset center is also used to build a user system; wherein the user system includes a permission system and an internal account system; Configure internal permissions for different users according to the permission system; According to the internal account system, account configuration is performed for user data sources, where account configuration includes: login factor configuration, high-risk operation factor configuration, and basic authentication factor configuration.

[0109] The principle of the above technical solution is: In the actual implementation process, a user center is set up for the construction of the user system, which supports users in managing the accounts and organizational structures of regulatory units, supports the function and data authorization of administrators and ordinary accounts, and realizes the establishment of an account system and authority system within the system.

[0110] The user list management platform supports the addition, deletion, enabling / disabling and authorization of accounts, and also supports the import of user information based on templates.

[0111] The organizational structure of users in the organizational structure management system supports the creation and management of branches.

[0112] Permission management can configure administrators or roles based on the permissions of their own login accounts. It supports configuration of functional permissions and data permissions for each module of the system based on system roles, and supports system user retrieval and corresponding role viewing.

[0113] The application list is mainly used to manage the identities of third-party applications and supports the setting of other configuration items of the application, such as login settings, permission settings, etc.

[0114] User settings support the configuration of user data sources, login factors, high-risk operation factors, basic authentication factors, account locking, and account login settings.

[0115] The personal center displays the basic information of the personal account and supports changing the password of the personal account.

[0116] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A network security monitoring and early warning method based on asset analysis, characterized in that: include: Build an asset center; the asset center is composed of an asset logic data model connected to user asset data; Based on Flink status monitoring, configure a security monitoring network for the asset center and determine the execution nodes and data nodes of the asset center. According to the execution nodes and data nodes, set up early warning analysis strategies for analysis and judgment; According to the early warning analysis strategy, analyze the user asset data to determine whether there is an early warning event.

2. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The said constructing asset center also includes: Pre-build the first fine-grained model of the asset overview; Through the first fine-grained model, configure the search conditions, query the asset information of any user, and obtain the asset aggregation information; According to the asset aggregation information, set up distributed asset reporting ports and asset automatic collection equipment to obtain the target asset data of any user; Construct a first network database of target asset data, perform asset data confidence calculations, and determine asset status.

3. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The configuration of a safety monitoring network for the asset center based on Flink status monitoring includes: Pre-build the Flink operating architecture and Flink programming model; According to the Flink operating architecture, a stream computing processing engine is built. The stream computing processing engine is used for stream processing and batch processing. According to the Flink programming model and streaming computing processing engine, asynchronous access to local state is stored and identified.

4. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The security monitoring network is configured with a log retrieval component, an alarm overview component, an alarm management component and a security event feedback component; The log retrieval component is used to perform real-time retrieval of the original log library to determine whether there is an alarm event. The real-time retrieval includes advanced mode and quick mode. In advanced mode, the query conditions entered by the user are used to retrieve alarm events. In quick mode, the user directly searches through predefined search conditions. The alarm overview component is used to count alarm events based on preset range conditions; The alarm management component is used to analyze the data source connected to the asset center and determine whether there is an alarm event; The security incident feedback component is used to perform closed-loop processing of key network security incidents through a preset key network security incident identification mechanism; wherein the closed-loop processing includes incident discovery, incident notification, incident rectification and incident feedback.

5. A network security monitoring and early warning method based on asset analysis as claimed in claim 4, characterized in that: Before determining the execution node and data node of the asset center, the method further includes: Build an interactive search engine based on a distributed non-relational database; According to the interactive search engine, the asset data of the asset center is classified according to the documents; According to data classification, event data that has passed data quality assessment is determined; wherein the event data includes normalized data fields and original logs, and the event data corresponds to an index file; According to the index file, the time data is sharded to generate a horizontal table; each shard of the horizontal table corresponds to a search engine, and each shard is configured with multiple copies.

6. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The execution node is used to receive the index request and calculate the shard into which the event document corresponding to the index request should be loaded; The data node is used to send the index request to the corresponding target node and generate inverted index information of the event document.

7. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: After determining the execution node and data node of the asset center, the method further includes: Establish analysis models, including multi-dimensional analysis models, attack chain analysis models, and high-risk IP analysis models; The multidimensional analysis model is used to obtain the traffic behavior of any user asset data in the data center, identify key entities, and determine each security risk event based on the target relationship network between key entities and attack events; Attack chain analysis is used to determine the attack source and victim address of the network attack. Based on the attack source and victim address, the attack process on user asset data is restored to identify high-risk events. High-risk IP analysis is used to purify the user asset data of the data center and determine whether an accident has occurred based on the purified high-quality small data.

8. The network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The early warning analysis strategy for setting up analysis and judgment includes: Obtain user access information supervised by the asset center and build a network security ecosystem based on access users; Configure the plan arrangement according to the network security ecosystem; the plan arrangement includes the work plan and the network security incident emergency plan; When a behavioral event triggers a plan, the early warning analysis strategy for the corresponding behavioral event is output.

9. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The asset center is also used to: Pre-set custom function interface and business data for business configuration; According to the functional interface, configure the content editor to edit and publish the data in the asset center; Build a business knowledge base based on business data; the business knowledge base includes the IP address library and the ATT&CK library; Generate a data warehouse based on standardized documents based on business knowledge base, data editing and data publishing.

10. A network security monitoring and early warning method based on asset analysis as claimed in claim 1, characterized in that: The asset center is also used to build a user system; wherein the user system includes a permission system and an internal account system; Configure internal permissions for different users according to the permission system; According to the internal account system, account configuration is performed for user data sources, where account configuration includes: login factor configuration, high-risk operation factor configuration, and basic authentication factor configuration.

Images