Network security risk assessment system based on big data analysis

Abstract

The invention relates to a network security risk assessment system based on big data analysis, which relates to the technical field of big data analysis and comprises a data acquisition module, a data storage and management module, a dynamic risk map construction and updating module, an intelligent risk association module and a risk assessment module. The data acquisition module is used for collecting various network security related data, and the related data comprises asset information, threat intelligence, vulnerability information and security event records in a network. According to the network security risk assessment system provided by the invention, asset information, threat intelligence, vulnerability information and security event records in a network are comprehensively collected through the data acquisition module, a rich and comprehensive data basis is provided for risk assessment, the acquired data are efficiently stored, and the risk assessment efficiency is improved through data cleaning, preprocessing, indexing and classification operations. The quality and availability of the data are improved, and accurate data support is provided for subsequent risk assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a network security risk assessment system based on big data analysis. Background Art

[0002] In the current network security field, with the rapid development of information technology and the widespread popularization of network applications, network security issues have become increasingly prominent and have become an important factor restricting the process of informatization. Traditional network security risk assessment methods often rely on manual experience and static data analysis, which is difficult to fully and real-time reflect the security status of network systems. Especially in the face of large-scale, complex and changeable network attacks, traditional methods often seem to be powerless and unable to timely and effectively identify, evaluate and respond to network security risks; Specifically, traditional methods can usually only collect limited network security related data, such as asset information, vulnerability information, etc., while ignoring key information such as threat intelligence and security incident records, resulting in inaccurate risk assessment results. Faced with massive network security data, traditional methods often lack effective data processing and analysis methods, and it is difficult to extract valuable information from the data, which in turn affects the efficiency and accuracy of risk assessment. Traditional methods are usually based on static risk assessment models, which cannot reflect the security status and risk change trends of network systems in real time, and it is difficult to meet the dynamic needs of current network security risk assessment. Traditional methods have shortcomings in correlation analysis, and it is difficult to discover the potential correlation between security incidents and the attack patterns behind them, which affects the depth and breadth of risk assessment. In this regard, we propose a network security risk assessment system based on big data analysis. Summary of the invention

[0003] In view of the technical problems existing in the prior art, the present invention provides a network security risk assessment system based on big data analysis.

[0004] The technical solution of the present invention to solve the above technical problems is as follows: A network security risk assessment system based on big data analysis, comprising: a data acquisition module, a data storage and management module, a dynamic risk map construction and update module, an intelligent risk association module, and a risk assessment module; The data collection module is used to collect various network security related data, including: asset information, threat intelligence, vulnerability information and security event records in the network; The data storage and management module is used to store network security data using big data storage technology, and to clean, pre-process, index and classify the stored network security data; The dynamic risk map construction and update module is used to construct a network security risk map based on the processed data, and to update the map in real time based on the newly collected data and predict the risk development trend; The intelligent risk association module is used to extract key features of security events, establish association rules, analyze the association between security events and output the association results intuitively; The risk assessment module is used to comprehensively assess network security risks.

[0005] Preferably, the data collection module includes: an asset data collection unit, a threat intelligence collection unit, a vulnerability data collection unit, and a security event data collection unit; The asset data collection unit is used to collect asset information of hardware devices in the network, and the asset information includes: model, configuration, location and software system version, function, and installation path; The threat intelligence collection unit is used to collect data including known network attack patterns, malware signatures, and activity signs of hacker organizations from a variety of external threat intelligence sources and internal security monitoring systems; The vulnerability data collection unit is used to collect vulnerability information of assets, scan network assets through vulnerability scanning tools to obtain software vulnerabilities and configuration error information, and receive vulnerability announcement information released by software vendors and security communities; The security event data collection unit collects security event records from security devices in the network. The security event records include: time, location, and event type information of the event.

[0006] Preferably, the data storage and management module includes: a big data storage unit, a data cleaning and preprocessing unit, and a data indexing and classification unit; The big data storage unit is used to store the collected network security data using big data storage technology; The data cleaning and preprocessing unit is used to clean the collected raw data, and the cleaning includes: removing duplicate data, correcting incorrect data formats, filling missing data values, and standardizing the data; The data indexing and classification unit is used to establish an indexing mechanism for stored data and to quickly locate and retrieve data through the index.

[0007] Preferably, the operations of cleaning, preprocessing, indexing and classifying the stored network security data specifically include: By comparing the key attributes of the data, use data structures such as hash tables to determine whether there is duplicate data. If duplicate data is found, keep one of them and delete the other duplicates; Define data format standards and convert data that does not meet the standards to correct incorrect data formats; Analyze the characteristics and correlation of data, and use the mean, median, or mode to fill in missing values ​​for numerical data; for categorical data, use the most frequently occurring category to fill in; For numerical data, the formula is used for normalization. The formula for normalization is:

[0008] In the formula, represents the normalized data, x represents the original data value, and Respectively represent the minimum and maximum values ​​in the data set; Establish an index based on the key attributes of the data, including: the IP address of the asset, the number of the vulnerability, and the timestamp of the security event. Classify the data according to its type, importance, and timestamp, create different database tables to store different types of data, and add classification tags or attributes to the data so that specific types of data can be quickly filtered out when querying.

[0009] Preferably, the dynamic risk map construction and update module includes: a map construction unit, a real-time update unit and a risk trend prediction unit; The graph construction unit is used to construct an initial network security risk graph based on the asset, threat, and vulnerability information processed by the data storage and management module. The elements such as assets, threats, and vulnerabilities are used as nodes of the graph, and the relationship between them is represented by edges. The graph database is used to realize graph construction. The real-time update unit is used to use big data analysis algorithms to process newly collected data in real time and dynamically update the risk map. When new assets are found online or old assets are offline, the asset nodes are updated in time. When new threats or vulnerabilities are detected, the corresponding nodes are added and the association between them and the assets is updated. When a security incident occurs, the risk status of the relevant nodes in the map is adjusted according to the nature and impact of the incident to ensure that the map can reflect the network security status in real time. The risk trend prediction unit is used to predict the development trend of risks by analyzing the changing trends of historical data and real-time data.

[0010] Preferably, a network security risk map is constructed based on the processed data, and the map is updated in real time based on the newly collected data, and the risk development trend is predicted, specifically including: Obtain processed asset, threat and vulnerability information from the data storage and management module; Determine the data model of the graph database, define the node types and edge relationship types, create assets, threats, and vulnerabilities as nodes, and create edges based on the actual relationships between them; Continuously receive newly collected data. When a new asset is detected online, create a new asset node in the graph and establish associations with the corresponding vulnerability nodes and threat nodes based on the possible vulnerabilities and threats it faces; When an old asset goes offline, delete the corresponding asset node and its related edges. When a new threat or vulnerability is detected, add a new threat node or vulnerability node and check whether the existing assets are affected by the new threat or vulnerability. If so, establish the corresponding edge. When a security incident occurs, analyze the nature and impact of the incident. If the incident involves a specific combination of assets, threats, and vulnerabilities, adjust the attributes of the relevant nodes and update the attributes of the edges to reflect the impact of the incident. Collect historical and real-time data, including asset vulnerability history, threat frequency, and time series of security events; The time series analysis method is used to analyze historical data, and the exponential smoothing method is used to predict the probability of vulnerability exploitation. The formula is:

[0011] In the formula, represents the predicted value for the next period, Indicates the current actual value. represents the current forecast value, represents the smoothing coefficient, where ; A prediction model is built based on LSTM to predict time series data. The characteristics of assets, threats, and historical security events are used as inputs, and the output is the predicted probability value of a certain asset being attacked by a specific threat. The expression of the prediction model is:

[0012] In the formula, Represents the prediction result, represents the weight matrix of the fully connected layer, It represents the hidden state at the last moment T after T time steps. represents the bias term; The prediction results are presented in a risk map.

[0013] Preferably, the intelligent risk association module includes: a feature extraction unit, an association rule establishment unit, an association analysis unit and an association result output unit; The feature extraction unit is used to extract key features from the security event data, wherein the features include: extracting the source IP, destination IP, port number, protocol type, event type and timestamp of the event; The association rule establishment unit is used to establish association rules between security events based on knowledge and experience in the field of network security; The correlation analysis unit is used to use big data correlation analysis technology to conduct correlation analysis on security events according to the established correlation rules. Whenever a new security event occurs, by comparing the characteristics of the event with the correlation rules, it automatically explores other events that are potentially related to the event and determines the degree of correlation between them, thereby discovering the correlation behind the security event; The correlation result output unit is used to present the results obtained from the correlation analysis in an intuitive manner and generate a correlation event report.

[0014] Preferably, establishing association rules and analyzing associations between security events specifically include: By organizing a team of cybersecurity experts, we sorted out a large number of past security incident cases and common attack patterns and security mechanism knowledge in the industry, summarized the correlation between different security incidents and collected domain knowledge and experience; The summarized association rules are expressed in the form of logical rules. If the same source IP continuously initiates connection requests to the same port of multiple different destination IPs, and the connection request frequency exceeds the threshold f, then the events are different manifestations of the same scanning behavior and are related; Select several historical security event data as test sets, apply established association rules to conduct association analysis, compare with actual known association situations, and verify the accuracy of the rules; The security event data based on the extracted key features are organized into a data table, where each row represents a security event and each column corresponds to a different key feature; Calculate the frequency of occurrence of each single feature, that is, the support, set the minimum support threshold, and filter out frequent 1-item sets that meet the threshold, that is, single feature sets with higher occurrence frequencies. The calculation formula for support is:

[0015] In the formula, represents the number of transactions containing item set X, Indicates the total number of transactions; Continuously combine frequent k-item sets to generate candidate k+1-item sets, recalculate the support, and filter out frequent k+1-item sets that meet the minimum support threshold; Repeat the above process until no new frequent item sets can be generated; Based on the obtained frequent item sets, association rules are generated according to the confidence index, and the rules that meet the minimum confidence threshold are screened out as valid association rules, where the confidence calculation formula is:

[0016] It represents the probability of B occurring when A occurs.

[0017] Preferably, the risk assessment module includes: a risk indicator determination unit, a risk value calculation unit and a correlation impact analysis unit; The risk indicator determination unit is used to determine various indicators for measuring network security risks based on relevant standards in the field of network security, past experience, and the characteristics of the analyzed network system; The risk value calculation unit is used to calculate the corresponding risk value for each asset, threat and vulnerability combination based on the determined risk indicators and various network security related data obtained by combining quantitative and qualitative methods; The correlation impact analysis unit is used to analyze the impact of correlation events on the overall risk assessment.

[0018] Preferably, the method for comprehensively assessing network security risks is: By consulting international standards, industry specifications and security guidelines issued by authoritative organizations related to cybersecurity, and combining the practical experience accumulated from handling cybersecurity incidents in the past, we have sorted out common indicator dimensions for measuring risks; Gain in-depth understanding of the business type, architecture characteristics, and data sensitivity of the network system being analyzed; Determine specific risk indicators through expert evaluation, classify them and describe them qualitatively, then convert them into corresponding quantitative scores and set weights for different indicators; The risk value is calculated by weighted average, where the weighted average calculation formula is:

[0019] The comprehensive risk value is obtained by summing up the product of each indicator score and the corresponding weight.

[0020] The beneficial effects of the present invention are: The network security risk assessment system proposed in the present invention comprehensively collects asset information, threat intelligence, vulnerability information and security event records in the network through the data acquisition module, provides a rich and comprehensive data basis for risk assessment, efficiently stores the collected data, and improves the quality and availability of data through data cleaning, preprocessing, indexing and classification operations, provides accurate data support for subsequent risk assessment, and updates the map in real time to reflect the latest status of network security, so that risk assessment can keep up with the dynamic changes of network security, improves the timeliness and accuracy of the assessment, and the intelligent risk association module can extract the key features of security events, establish association rules, and analyze the association between security events, which is helpful to discover the potential connections and attack patterns behind security events, and provide strong support for network security protection. The risk value is calculated by combining quantitative and qualitative methods, and the impact of associated events on the overall risk assessment is analyzed, so that the assessment results are more comprehensive and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a module structure diagram of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0023] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0024] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0025] Reference Figure 1 As shown, a network security risk assessment system based on big data analysis includes: 1. Data acquisition module Asset Data Collection Unit Collect hardware equipment asset information on the network, including servers, computers, network devices, etc. Cover model, configuration, location, software system version and function information, and installation path.

[0026] Threat Intelligence Collection Unit Collect information from external threat intelligence sources and internal security monitoring systems, including known attack patterns, malware signatures, and signs of hacker group activity.

[0027] Vulnerability Data Collection Unit Obtain software vulnerabilities and configuration errors of network assets through vulnerability scanning tools, and receive vulnerability announcement information released by software vendors and security communities.

[0028] Security event data collection unit Collect security event records from network security devices, including time, location, and event type information.

[0029] 2. Data Storage and Management Module Big data storage unit Big data storage technology is used to store network security data, which has high reliability, high scalability and high performance.

[0030] Data cleaning and preprocessing unit Clean and preprocess the raw data, including removing duplicate data, correcting incorrect formats, filling missing values, and performing standardization.

[0031] Data index and classification unit Establish an indexing mechanism for stored data to improve retrieval efficiency and classify data according to asset type, threat type, vulnerability type, etc.

[0032] 3. Dynamic Risk Map Construction and Update Module Graph building unit An initial network security risk map is constructed based on the processed asset, threat, and vulnerability information. Assets, threats, and vulnerabilities are used as nodes, and edges are used to represent associations. The map is constructed using a graph database.

[0033] Real-time update unit Utilize big data analysis algorithms to process new data in real time and update risk maps, including updating asset nodes, adding new threat or vulnerability nodes and updating associations, and adjusting the risk status of nodes related to security incidents.

[0034] Risk Trend Forecasting Unit By analyzing the changing trends of historical data and real-time data, we can predict the development trend of risks and use machine learning algorithms and data analysis techniques to find out the patterns of risk changes.

[0035] 4. Intelligent Risk Association Module Feature Extraction Unit Extract key features such as source IP, destination IP, port number, protocol type, event type and timestamp from security event data.

[0036] Association rule building unit Establish correlation rules between security events based on network security knowledge and experience.

[0037] Correlation Analysis Unit Utilize big data correlation analysis technology to conduct correlation analysis on security events according to correlation rules and determine the degree of correlation.

[0038] Correlation result output unit The correlation analysis results are presented in an intuitive way, and a correlation event report is generated, including security event details, correlation rule descriptions, correlation event lists, and correlation levels.

[0039] 5. Risk Assessment Module Risk indicator determination unit Based on network security standards, experience and system characteristics, determine indicators for measuring network security risks, including asset value, threat level, vulnerability severity and impact of security incidents.

[0040] Value at Risk Calculation Unit A combination of quantitative and qualitative methods is used to calculate the risk value of each asset, threat and vulnerability combination based on risk indicators and network security data.

[0041] Correlation Impact Analysis Unit Analyze the impact of related events on the overall risk assessment, and evaluate the nature and degree of correlation of related events and their impact on risk indicators.

[0042] The cleaning, preprocessing, indexing and classification operations of stored network security data specifically include: By comparing the key attributes of the data, use data structures such as hash tables to determine whether there is duplicate data. If duplicate data is found, keep one of them and delete the other duplicates; Define data format standards and convert data that does not meet the standards to correct incorrect data formats; Analyze the characteristics and correlation of data, and use the mean, median, or mode to fill in missing values ​​for numerical data; for categorical data, use the most frequently occurring category to fill in; For numerical data, the formula is used for normalization. The formula for normalization is:

[0043] In the formula, represents the normalized data, x represents the original data value, and Respectively represent the minimum and maximum values ​​in the data set; Establish an index based on the IP address of the asset, the number of the vulnerability, and the timestamp of the security event. At the same time, create different database tables to store asset, vulnerability, and security event data based on data type, importance, and timestamp classification, and add classification labels or attributes to the data.

[0044] Obtain asset, threat, and vulnerability information from the data storage and management module, determine the graph database data model, define the node type and edge relationship type, create assets, threats, and vulnerabilities as nodes, and create edges based on the actual relationships.

[0045] Continuously receive new data. When new assets come online, nodes are created and associated with vulnerability and threat nodes. When old assets go offline, nodes and edges are deleted. When new threats or vulnerabilities appear, nodes are added and the impact on assets is checked to establish edges. When security incidents occur, the nature and impact are analyzed to adjust node and edge properties.

[0046] Collect historical and real-time data, including asset vulnerability history, threat frequency, and security event time series, to analyze and understand risk development trends and take preventive measures.

[0047] The time series analysis method is used to analyze historical data, and the exponential smoothing method is used to predict the probability of vulnerability exploitation. The formula is:

[0048] In the formula, represents the predicted value for the next period, Indicates the current actual value. represents the current forecast value, represents the smoothing coefficient, where ; A prediction model is built based on LSTM to predict time series data. The characteristics of assets, threats, and historical security events are used as inputs, and the output is the predicted probability value of a certain asset being attacked by a specific threat. The expression of the prediction model is:

[0049] In the formula, Represents the prediction result, represents the weight matrix of the fully connected layer, It represents the hidden state at the last moment T after T time steps. represents the bias term; The prediction results are presented in a risk map.

[0050] Establishing association rules and analyzing the association between security events specifically include: Organize a team of network security experts to conduct an in-depth review of a large number of past security incident cases, and comprehensively study common attack patterns and security mechanism-related knowledge in the industry. In this way, we can collect rich domain knowledge and experience, and then summarize the correlation between different security incidents. For example, we may find that certain types of attacks are more likely to occur in specific network environments or system configurations, or that the occurrence of certain security incidents often indicates an increased likelihood of other specific security incidents in the future; Clearly express the summarized association rules in the form of logical rules. For example, if within a specific time interval, the same source IP continuously initiates connection requests to the same port of multiple different destination IPs, and the frequency of connection requests exceeds the preset threshold f, then it can be determined that these events are different manifestations of the same scanning behavior and there is a correlation between them. Such logical rules can help us quickly and accurately identify security events with correlations; Select several historical security event data as the test set, and apply the established association rules to perform association analysis on these data. Compare the associations obtained by analysis with the actual known associations to verify the accuracy of the rules. If the analysis results are highly consistent with the actual situation, it means that the rules are relatively accurate and reliable; if there are large differences, the rules need to be further adjusted and optimized; The security event data based on the extracted key features are organized into a data table. In the data table, each row represents a security event, and each column corresponds to a different key feature. For example, it may include the time when the event occurred, the source IP address, the destination IP address, the port number, the attack type, and other features; Calculate the frequency of occurrence of each single feature, that is, the support, set the minimum support threshold, and filter out frequent 1-item sets that meet the threshold, that is, single feature sets with higher occurrence frequencies. The calculation formula for support is:

[0051] In the formula, represents the number of transactions containing item set X, Indicates the total number of transactions; Continuously combine frequent k-item sets to generate candidate k+1-item sets, recalculate the support, and filter out frequent k+1-item sets that meet the minimum support threshold; Repeat the above process until no new frequent item sets can be generated; Based on the obtained frequent item sets, association rules are generated according to the confidence index, and the rules that meet the minimum confidence threshold are screened out as valid association rules, where the confidence calculation formula is:

[0052] It represents the probability of B occurring when A occurs.

[0053] The method for comprehensively assessing cybersecurity risks is: By extensively reviewing international standards, industry specifications and security guidelines issued by authoritative organizations related to network security. In this process, you can conduct in-depth research on network security management standards such as those developed by the International Organization for Standardization (ISO), industry-specific network security specifications, and security guidelines issued by authoritative organizations such as the International Computer Security Association. These standards, specifications and guidelines usually cover all aspects of network security, including but not limited to risk management, security policy formulation, technical protection measures, etc. At the same time, combined with the rich practical experience accumulated from handling network security incidents in the past. Review the various network security incidents that have been handled, analyze the causes, development process, response measures taken and the final results of the incidents. By summarizing and reflecting on these experiences, we can better understand different types of network security risks and effective response methods. Combining the above two aspects, sort out the common indicator dimensions for measuring risks. These common indicator dimensions may include but are not limited to asset value, threat possibility, vulnerability level, and security control effectiveness; Gain in-depth understanding of the business type of the network system being analyzed. Different business types may have very different requirements and risk points for network security. For example, financial business may have extremely high requirements for data confidentiality and integrity, while e-commerce business may be more concerned with transaction security and user privacy protection. It is also crucial to understand the architectural characteristics of the network system. This includes aspects such as network topology, server configuration, and application architecture. Different architectural characteristics will affect the distribution and propagation of security risks. In addition, it is necessary to understand the sensitivity of the data. Determine which data is sensitive, such as personal identity information, financial data, business secrets, etc., and how these sensitive data are stored, transmitted, and processed in the network system; Determine specific risk indicators through expert evaluation. Invite experts in the field of network security to evaluate the risks of network systems based on their professional knowledge and experience. Classify these risk indicators, for example, they can be divided into technical risk indicators, management risk indicators, human risk indicators, etc. Qualitatively describe each risk indicator to explain its meaning and scope of impact. Then, convert the qualitative description into a corresponding quantitative score. For example, a quantitative method such as a five-point system or a ten-point system can be used to assign a specific score to each risk indicator. At the same time, set weights for different indicators. The setting of weights can be determined based on the importance of the indicator, the degree of impact, and the role of the indicator in network security risk assessment. For example, a higher weight may be given to the availability indicator of key business systems, while a lower weight may be given to some relatively minor indicators; The risk value is calculated by weighted average, where the weighted average calculation formula is:

[0054] The comprehensive risk value is obtained by summing up the product of each indicator score and the corresponding weight.

[0055] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0056] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0057] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A network security risk assessment system based on big data analysis, characterized in that: include: Data collection module, data storage and management module, dynamic risk map construction and update module, intelligent risk association module, risk assessment module; The data collection module is used to collect various network security related data, including: asset information, threat intelligence, vulnerability information and security event records in the network; The data storage and management module is used to store network security data using big data storage technology, and to clean, pre-process, index and classify the stored network security data; The dynamic risk map construction and update module is used to construct a network security risk map based on the processed data, and to update the map in real time based on the newly collected data and predict the risk development trend; The intelligent risk association module is used to extract key features of security events, establish association rules, analyze the association between security events and output the association results intuitively; The risk assessment module is used to comprehensively assess network security risks.

2. A network security risk assessment system based on big data analysis according to claim 1, characterized in that: The data collection module includes: asset data collection unit, threat intelligence collection unit, vulnerability data collection unit and security event data collection unit; The asset data collection unit is used to collect asset information of hardware devices in the network, and the asset information includes: model, configuration, location and software system version, function, and installation path; The threat intelligence collection unit is used to collect data including known network attack patterns, malware signatures, and activity signs of hacker organizations from a variety of external threat intelligence sources and internal security monitoring systems; The vulnerability data collection unit is used to collect vulnerability information of assets, scan network assets through vulnerability scanning tools to obtain software vulnerabilities and configuration error information, and receive vulnerability announcement information released by software vendors and security communities; The security event data collection unit collects security event records from security devices in the network. The security event records include: time, location, and event type information of the event.

3. According to the network security risk assessment system based on big data analysis according to claim 1, it is characterized in that: The data storage and management module includes: a big data storage unit, a data cleaning and preprocessing unit, and a data indexing and classification unit; The big data storage unit is used to store the collected network security data using big data storage technology; The data cleaning and preprocessing unit is used to clean the collected raw data, and the cleaning includes: removing duplicate data, correcting incorrect data formats, filling missing data values, and standardizing the data; The data indexing and classification unit is used to establish an indexing mechanism for stored data and to quickly locate and retrieve data through the index.

4. A network security risk assessment system based on big data analysis according to claim 3, characterized in that: The cleaning, preprocessing, indexing and classification operations of stored network security data specifically include: By comparing the key attributes of the data, use data structures such as hash tables to determine whether there is duplicate data. If duplicate data is found, keep one of them and delete the other duplicates; Define data format standards and convert data that does not meet the standards to correct incorrect data formats; Analyze the characteristics and correlation of data, and use the mean, median, or mode to fill in missing values ​​for numerical data; for categorical data, use the most frequently occurring category to fill in; For numerical data, the formula is used for normalization. The formula for normalization is: In the formula, represents the normalized data, x represents the original data value, and Respectively represent the minimum and maximum values ​​in the data set; Establish an index based on the key attributes of the data, including: the IP address of the asset, the number of the vulnerability, and the timestamp of the security event. Classify the data according to its type, importance, and timestamp, create different database tables to store different types of data, and add classification tags or attributes to the data so that specific types of data can be quickly filtered out when querying.

5. The network security risk assessment system based on big data analysis according to claim 1 is characterized in that: The dynamic risk map construction and update module includes: map construction unit, real-time update unit and risk trend prediction unit; The graph construction unit is used to construct an initial network security risk graph based on the asset, threat, and vulnerability information processed by the data storage and management module. The elements such as assets, threats, and vulnerabilities are used as nodes of the graph, and the relationship between them is represented by edges. The graph database is used to realize graph construction. The real-time update unit is used to use big data analysis algorithms to process newly collected data in real time and dynamically update the risk map. When new assets are found online or old assets are offline, the asset nodes are updated in time. When new threats or vulnerabilities are detected, the corresponding nodes are added and the association between them and the assets is updated. When a security incident occurs, the risk status of the relevant nodes in the map is adjusted according to the nature and impact of the incident to ensure that the map can reflect the network security status in real time. The risk trend prediction unit is used to predict the development trend of risks by analyzing the changing trends of historical data and real-time data.

6. The network security risk assessment system based on big data analysis according to claim 1 is characterized in that: Build a network security risk map based on the processed data, and update the map in real time based on the newly collected data and predict the risk development trend. Specifically include: Obtain processed asset, threat and vulnerability information from the data storage and management module; Determine the data model of the graph database, define the node types and edge relationship types, create assets, threats, and vulnerabilities as nodes, and create edges based on the actual relationships between them; Continuously receive newly collected data. When a new asset is detected online, create a new asset node in the graph and establish associations with the corresponding vulnerability nodes and threat nodes based on the possible vulnerabilities and threats it faces; When an old asset goes offline, delete the corresponding asset node and its related edges. When a new threat or vulnerability is detected, add a new threat node or vulnerability node and check whether the existing assets are affected by the new threat or vulnerability. If so, establish the corresponding edge. When a security incident occurs, analyze the nature and impact of the incident. If the incident involves a specific combination of assets, threats, and vulnerabilities, adjust the attributes of the relevant nodes and update the attributes of the edges to reflect the impact of the incident. Collect historical and real-time data, including asset vulnerability history, threat frequency, and time series of security events; The time series analysis method is used to analyze historical data, and the exponential smoothing method is used to predict the probability of vulnerability exploitation. The formula is: In the formula, represents the predicted value for the next period, Indicates the current actual value. represents the current forecast value, represents the smoothing coefficient, where ; A prediction model is built based on LSTM to predict time series data. The characteristics of assets, threats, and historical security events are used as inputs, and the output is the predicted probability value of a certain asset being attacked by a specific threat. The expression of the prediction model is: In the formula, Represents the prediction result, represents the weight matrix of the fully connected layer, It represents the hidden state at the last moment T after T time steps. represents the bias term; The prediction results are presented in a risk map.

7. The network security risk assessment system based on big data analysis according to claim 1 is characterized in that: The intelligent risk association module includes: a feature extraction unit, an association rule establishment unit, an association analysis unit and an association result output unit; The feature extraction unit is used to extract key features from the security event data, wherein the features include: extracting the source IP, destination IP, port number, protocol type, event type and timestamp of the event; The association rule establishment unit is used to establish association rules between security events based on knowledge and experience in the field of network security; The correlation analysis unit is used to use big data correlation analysis technology to conduct correlation analysis on security events according to the established correlation rules. Whenever a new security event occurs, by comparing the characteristics of the event with the correlation rules, it automatically explores other events that are potentially related to the event and determines the degree of correlation between them, thereby discovering the correlation behind the security event; The correlation result output unit is used to present the results obtained from the correlation analysis in an intuitive manner and generate a correlation event report.

8. The network security risk assessment system based on big data analysis according to claim 1 is characterized in that: Establishing association rules and analyzing the association between security events specifically include: By organizing a team of cybersecurity experts, we sorted out a large number of past security incident cases and common attack patterns and security mechanism knowledge in the industry, summarized the correlation between different security incidents and collected domain knowledge and experience; The summarized association rules are expressed in the form of logical rules. If the same source IP continuously initiates connection requests to the same port of multiple different destination IPs, and the connection request frequency exceeds the threshold f, then the events are different manifestations of the same scanning behavior and are related; Select several historical security event data as test sets, apply established association rules to conduct association analysis, compare with actual known association situations, and verify the accuracy of the rules; The security event data based on the extracted key features are organized into a data table, where each row represents a security event and each column corresponds to a different key feature; Calculate the frequency of occurrence of each single feature, that is, the support, set the minimum support threshold, and filter out frequent 1-item sets that meet the threshold, that is, single feature sets with higher occurrence frequencies. The calculation formula for support is: In the formula, represents the number of transactions containing item set X, Indicates the total number of transactions; Continuously combine frequent k-item sets to generate candidate k+1-item sets, recalculate the support, and filter out frequent k+1-item sets that meet the minimum support threshold; Repeat the above process until no new frequent item sets can be generated; Based on the obtained frequent item sets, association rules are generated according to the confidence index, and the rules that meet the minimum confidence threshold are screened out as valid association rules, where the confidence calculation formula is: It represents the probability of B occurring when A occurs.

9. The network security risk assessment system based on big data analysis according to claim 1 is characterized in that: The risk assessment module includes: a risk indicator determination unit, a risk value calculation unit and a correlation impact analysis unit; The risk indicator determination unit is used to determine various indicators for measuring network security risks based on relevant standards in the field of network security, past experience, and the characteristics of the analyzed network system; The risk value calculation unit is used to calculate the corresponding risk value for each asset, threat and vulnerability combination based on the determined risk indicators and various network security related data obtained by combining quantitative and qualitative methods; The correlation impact analysis unit is used to analyze the impact of correlation events on the overall risk assessment.

10. The network security risk assessment system based on big data analysis according to claim 1 is characterized in that: The method for comprehensively assessing cybersecurity risks is: By consulting international standards, industry specifications and security guidelines issued by authoritative organizations related to cybersecurity, and combining the practical experience accumulated from handling cybersecurity incidents in the past, we have sorted out common indicator dimensions for measuring risks; Gain in-depth understanding of the business type, architecture characteristics, and data sensitivity of the network system being analyzed; Determine specific risk indicators through expert evaluation, classify them and describe them qualitatively, then convert them into corresponding quantitative scores and set weights for different indicators; The risk value is calculated by weighted average, where the weighted average calculation formula is: The comprehensive risk value is obtained by summing up the product of each indicator score and the corresponding weight.

Images