Security risk assessment system and method for digital archives

Abstract

The invention discloses a security risk assessment system and method for digital archives, and particularly relates to the technical field of security risk assesses.The security risk assessment method comprises the steps that firstly, relevant data of the archives are collected, and the collected data are preprocessed; then, performing preliminary analysis on the archive features, preliminarily judging abnormal feature weights, and sorting the abnormal features; different abnormal features are weighted, each abnormal feature weight is obtained by combining an entropy evaluation method and a CRITIC method, the abnormal feature weights are determined, timeliness and abnormal feature data volume are calculated through weighted summation to obtain a coincidence coefficient, and the coincidence coefficient is determined; and judging whether to use an isolated forest algorithm or a clustering analysis algorithm to further analyze the threat related characteristics by comparing the coincidence coefficient with a threshold value. Finally, after the content of the threat report is determined, the distribution and trend of the risk are visually displayed in a chart or instrument panel mode.

Description

Technical Field

[0001] The present invention relates to the technical field of security risk assessment, and more specifically, to a security risk assessment system and method for digital archives. Background Art

[0002] With the rapid development of information technology, digital archives are increasingly used in various fields. However, the security of digital archives faces unprecedented challenges. Problems such as cyber attacks, data leaks, malware attacks, and improper operations by internal personnel are becoming increasingly serious, bringing tremendous pressure to the security management of digital archives.

[0003] At present, traditional security measures such as firewalls, intrusion detection systems and encryption technology can prevent external threats to a certain extent, but they are powerless in the complex and ever-changing network security environment. They cannot adapt to complex environments, which leads to misjudgment of threat detection. In recent years, with the development of artificial intelligence and big data technology, intelligent security risk assessment systems have gradually become a research hotspot. These systems can monitor and analyze multi-dimensional data such as user behavior, system vulnerabilities, and network traffic in real time through statistical analysis and machine learning algorithms, identify behaviors that deviate from normal patterns, and thus achieve more accurate risk assessment. However, the existing systems still have some shortcomings, such as high computational costs and lack of dynamic adaptability, which limit their promotion and effectiveness in practical applications. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide a security risk assessment system and method for digital archives to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A security risk assessment method for digital archives comprises the following steps: Conduct a preliminary analysis of the archive features, preliminarily determine the weights of abnormal features, and organize the abnormal features; weight different abnormal features, and each abnormal feature weight is obtained by combining the entropy method and the CRITIC method; Determine the weight, timeliness, and amount of abnormal feature data of the abnormal features, calculate the compliance coefficient by weighted summation, and compare the compliance coefficient with the threshold to determine whether to use the isolation forest algorithm or cluster analysis algorithm to further analyze the threat-related features.

[0006] In a preferred embodiment, the abnormal features are sorted, including cleaning the abnormal features and counting the amount of abnormal feature data.

[0007] In a preferred embodiment, different abnormal features are weighted. Specifically, for the abnormal features, the difference coefficient is first calculated using the entropy method, and then the correlation is calculated using the CRITIC method; the weight of the abnormal feature is calculated using weighted summation based on the difference coefficient and the correlation.

[0008] In a preferred embodiment, after a preliminary analysis of the archive features is performed and abnormal features are screened out, the abnormal feature weights, timeliness, and amount of abnormal feature data are determined, and the compliance coefficient is calculated by weighted summation. In a preferred embodiment, the numerical value of the compliance coefficient is determined. If the numerical value is greater than the system threshold, the isolation forest algorithm is used to further analyze the abnormal features. If the numerical value is less than the system threshold, the clustering analysis algorithm is selected to further analyze the abnormal features.

[0009] In a preferred embodiment, the evaluation system comprises: The screening module is used to conduct a preliminary analysis of the file features, preliminarily determine the weight of abnormal features, and count the amount of abnormal feature data; The weighting module weights different abnormal features. The weight of each abnormal feature is obtained by combining the entropy method and the CRITIC method. The evaluation module is used to further analyze the abnormal characteristics and determine the specific threat situation.

[0010] In a preferred embodiment, the evaluation module includes a selection module, and the selection module is used to select an isolation forest module or a cluster analysis module to further analyze the abnormal features according to the abnormal feature weight, timeliness, and abnormal feature data volume.

[0011] Technical effects and advantages of the present invention: The present invention first collects archive-related data and pre-processes the collected data. Subsequently, the archive features are preliminarily analyzed, the abnormal feature weights are preliminarily determined, and the abnormal features are sorted; different abnormal features are weighted, and each abnormal feature weight is obtained by combining the entropy method and the CRITIC method, and the abnormal feature weight, timeliness, and abnormal feature data volume are determined by weighted summation to calculate the compliance coefficient, and the comparison of the compliance coefficient with the threshold is used to determine whether to use the isolation forest algorithm or the cluster analysis algorithm to further analyze the threat-related features. Finally, after determining the content of the threat report, the distribution and trend of the risk are intuitively displayed in the form of a chart or dashboard, and the data generated during the operation of the system is saved for subsequent reference and analysis. This makes the dynamic adaptability higher and the computational cost lower. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings; Figure 1 A schematic diagram of a flow chart of a security risk assessment method for digital archives according to the present invention; Figure 2 The present invention is a schematic structural diagram of a security risk assessment system for digital archives. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The present invention first collects archive-related data and pre-processes the collected data. Subsequently, the archive features are preliminarily analyzed, the abnormal feature weights are preliminarily determined, and the abnormal features are sorted; different abnormal features are weighted, and each abnormal feature weight is obtained by combining the entropy method and the CRITIC method, and the abnormal feature weight, timeliness, and abnormal feature data volume are determined by weighted summation to calculate the compliance coefficient, and the comparison of the compliance coefficient with the threshold is used to determine whether to use the isolation forest algorithm or the cluster analysis algorithm to further analyze the threat-related features. Finally, after determining the content of the threat report, the distribution and trend of the risk are intuitively displayed in the form of a chart or dashboard, and the data generated during the operation of the system is saved for subsequent reference and analysis. Example 1 The present invention provides a security risk assessment method for digital archives, such as Figure 1 As shown, it mainly includes the following steps: Step S1: Collecting archive-related data, and preprocessing the collected archive-related data; and extracting features from the processed archive-related data; Step S2: Use an abnormality detection method based on statistical process control to perform a preliminary analysis on the features and preliminarily determine the abnormal feature weights; sort out the abnormal features and weight the abnormal features; Step S3: Determine the abnormal feature weight, timeliness, and abnormal feature data volume, calculate the compliance coefficient by weighted summation, and determine whether to use the isolation forest algorithm or cluster analysis algorithm to further analyze the threat-related features by comparing the compliance coefficient with the threshold; generate a threat report; Step S4: Determine the threat report and use charts or dashboards to visually display risk distribution and trends; save the data generated during system operation.

[0015] In step S1, archive-related data includes but is not limited to archive storage location, access logs, permission settings, network traffic, etc. First, identify the type of data that needs to be collected. Each type of data has its own specific format and storage location. Then determine the frequency and level of detail of data collection. For example, access logs may be collected in real time, while network traffic may be summarized hourly. The management personnel can set it according to their needs. Finally, define the storage format of the data, such as CSV, JSON or database table structure, for subsequent processing and analysis.

[0016] Identify data sources: Server logs: locate the location of server log files; Database records: confirm the database table structure and field information for storing archive information; User operation records: obtain log information of user login, query, download and other operations; Network traffic: obtain relevant traffic data through network packet capture tools.

[0017] Access data sources: Use log management tools to centrally manage server logs; use database connection tools to access database records; configure network monitoring tools to capture network traffic data.

[0018] After obtaining the archive-related data, the archive-related data is preprocessed, which specifically includes the following steps: Step A1: Clean the archive-related data, remove noise data, and ensure the integrity and consistency of the data; Step A2: Convert the archive-related data and format the data from different sources in a unified way to facilitate subsequent analysis.

[0019] The specific step A1 also includes the following steps: Step A1.1: Use a hash table or database uniqueness constraint to detect duplicate records; identify duplicate entries by checking fields such as timestamp, IP address, and operation type in the log; then use SQL statements or programming languages ​​to delete duplicate records, and record the number of duplicate records deleted and the reason for deletion for subsequent review; Step A1.2: Identify invalid logs and check whether the log format is correct; use regular expressions to verify whether the log content conforms to the expected format; then delete the log entries with incorrect format or cannot be parsed; at the same time, establish log quality standards and regularly check the validity of the logs; Step A1.3: First, convert all timestamps to a unified format. Then, use tools such as Python's pytz library to convert timestamps in different time zones to a unified time zone. Finally, check whether the timestamps have jumps or large intervals to ensure the continuity of the time series. Step A1.4: Fill missing fields, use default values ​​to fill missing data or use interpolation methods to fill missing values; at the same time, mark missing data and mark missing fields in the data; Step A1.5: Filter the abnormal data. First, use statistical methods to identify abnormal values. Then remove the data that obviously does not conform to the business logic, delete negative numbers, excessively high values, or other unreasonable values. At the same time, save the removed abnormal data into a separate log file for subsequent analysis.

[0020] The specific step A2 includes the following steps: Step A2.1: Convert data from different sources (such as access logs, network traffic, and system logs) into a unified format; and convert field names to ensure consistent field naming; Step A2.2: Standardize the numerical data first; use the Min-Max standardization formula: ; Step A2.3: Encode the categorical data and perform preprocessing operations such as word segmentation and stop word removal on the text field; Step A2.4: Convert the timestamp to a unified time format; aggregate the data by time period, such as hour, day, week, etc.

[0021] Feature extraction is the core link in the risk identification and analysis module. Its goal is to extract key features from preprocessed data for subsequent threat identification and vulnerability detection. The following are the detailed steps: Step B1: Extract key features from user behavior data. For small and medium-sized data, the Python ecosystem is the first choice; for large-scale data, Spark or Hadoop is needed. Identify abnormal behavior patterns; Login time distribution: Count the user's login time distribution and analyze whether there is an abnormal login pattern (such as frequent logins during non-working hours). Login frequency: Calculate the number of user logins per unit time and identify high-frequency login behaviors. Access resource type: Extract the resource type accessed by the user (such as documents, pictures, videos) and analyze the access pattern. IP address changes: Analyze the changes in the user's login IP address. Count the user's access frequency and identify abnormal fluctuations; extract the number and type of resources accessed by the user.

[0022] Step B2: Extract key features from system operation data to identify potential security vulnerabilities; system monitoring tools are used to collect and store data; security log analysis tools are used to identify abnormal behavior; CPU usage: extract the system's CPU usage data and analyze whether there are abnormal fluctuations; memory usage: analyze the system's memory usage and identify memory leaks or abnormal usage; disk I / O: extract the frequency and size of disk read and write operations and analyze whether there are abnormal activities; extract the list of processes running in the system and identify unknown or suspicious processes; analyze the behavioral characteristics of the processes.

[0023] Step B3: Extract key features from network traffic data to identify potential network threats. Use the Wireshark tool to obtain; Packet size: extract the packet size distribution in network traffic and identify abnormal patterns; Transport protocol: analyze the transport protocol used in network traffic; Connection duration: extract the duration of network connection and identify abnormal connection behavior; Analyze the destination IP address and port of the network connection to identify abnormal connection behavior; Count the frequency and pattern of connections per unit time.

[0024] In step S2, the features are firstly analyzed using an anomaly detection method based on statistical process control, which specifically includes the following steps: Step C1: Build a model based on statistical methods to identify abnormal behavior patterns; detect abnormal fluctuations in time series; calculate the mean, variance, standard deviation and other statistics of the time series, and use the moving average method, exponential smoothing method or ARIMA model to predict normal behavior patterns; compare the deviation between actual behavior and predicted behavior to identify abnormal fluctuations.

[0025] Identify abnormal samples through statistical hypothesis testing; set the null hypothesis and alternative hypothesis; select the test method: use t-test, chi-square test and other methods to test the hypothesis; determine whether to reject the null hypothesis based on the test results.

[0026] Establish a relationship model between variables to predict potential threats; use linear regression or logistic regression to analyze the relationship between features and threats; train regression models based on historical data; use models to predict potential threats and analyze the impact of features on threats; Identify abnormal behavior based on probability distribution; assume that normal behavior follows a certain probability distribution; calculate the probability density or cumulative distribution function value of the sample; detect behavior that exceeds the confidence interval.

[0027] Step C2: Identify abnormal behavior based on the results of the statistical model; identify samples that deviate from the normal range; calculate the mean and variance of the feature to identify samples that deviate from the normal range; calculate the quantile of the feature to identify samples that are out of range.

[0028] Quantify the degree of abnormality of the sample; calculate the Z-score of each sample through Z-score calculation to identify the behavior that deviates from the mean. Identify outliers that are beyond the interquartile range through box plots.

[0029] Record abnormal samples and their causes; mark abnormal samples according to statistical results; Step C3: Interpret the results of the anomaly detection method based on statistical process control; classify threats according to anomaly characteristics; classify threats according to abnormal behavior patterns.

[0030] Through the above steps, statistical modeling and anomaly detection methods can be systematically used to identify potential threats. From time series analysis to hypothesis testing, to regression analysis and probability distribution modeling, each step provides a solid theoretical basis for threat detection and preliminarily determines the weight of abnormal features.

[0031] After identifying the abnormal feature weights, the collected abnormal features are collected and the amount of abnormal feature data is counted.

[0032] For abnormal features, the entropy method is first used to calculate the difference coefficient xs, and then the CRITIC method is used to calculate the correlation gs; the abnormal feature weight is obtained by weighted summation, and the specific formula is as follows: Q=δxs+εgs; where Q is the abnormal feature weight, δ and ε are the weight coefficients of the difference coefficient and the correlation, respectively.

[0033] In step S3, when statistical analysis detects significant anomalies, the abnormal feature weight, timeliness, and amount of abnormal feature data are determined to determine whether to use the isolation forest algorithm or the cluster analysis algorithm to further detect the abnormal feature weight.

[0034] Timeliness plays a vital role in archive threat detection. The Isolation Forest algorithm has an advantage in scenarios that require rapid response due to its efficient real-time processing capabilities and low resource consumption; while the Cluster Analysis algorithm excels in discovering potential patterns in data and is suitable for offline analysis. Timeliness is set by the staff.

[0035] The abnormal feature weight, timeliness, and abnormal feature data volume are normalized; the compliance coefficient is calculated using the weighted sum formula, which is as follows: F=αxn+βsx+γgm; where α, β, and γ are weight coefficients of abnormal feature weight, timeliness, and abnormal feature data volume; xn is the abnormal feature weight; sx is the timeliness; and gm is the abnormal feature data volume.

[0036] The algorithm is selected according to the numerical value of the compliance coefficient. If the value is greater than the system threshold, the isolation forest algorithm is used, otherwise the cluster analysis is selected.

[0037] Isolation Forest: Suitable for scenarios that directly detect anomalies, such as network intrusion detection, malware identification, etc.; especially suitable for anomaly detection of high-dimensional data and large data sets; suitable for real-time and efficient scenarios because of its low time complexity (O(n log n)).

[0038] Cluster analysis: Applicable to scenarios that require clear grouping, such as user behavior classification, system log grouping, etc.; can discover natural grouping structures in the data and help identify different types of threats; suitable for exploratory data analysis and can reveal potential patterns in the data.

[0039] The algorithm is selected based on the numerical value of the compliance coefficient. If the value is greater than the system threshold, the isolation forest algorithm is used. The steps for using the isolation forest threat detection are as follows: Step D1: Identify threat-related features and divide the data into training and test sets (e.g., 70% training set + 30% test set); ensure that the training set contains enough normal samples; Step D2: Build the tree model and initialize the parameters; set the key parameters of the isolation forest: n_estimators: the number of isolation trees; max_samples: the sample ratio used by each tree; random_state: random seed to ensure repeatable results; Contamination: The expected proportion of abnormal samples, set based on business experience; Step D3: Train the model and use the training set data to train the isolation forest model. The model constructs multiple isolation trees by randomly splitting the data. The structure of each tree determines the isolation depth of the sample.

[0040] Step D4: Use the trained model to identify potential threats; calculate the anomaly score for each sample in the test set, which reflects the degree of abnormality of the sample; the anomaly analysis calculation formula is as follows: ; x represents the sample to be detected; Average Depth(x) represents the average isolation depth of sample x in all isolation trees in the isolation forest. The isolation depth refers to the number of splits required to isolate sample x. Outliers usually have a smaller isolation depth because they are easier to be isolated; Expected Depth represents the expected depth when randomly splitting a complete binary tree with a height of h under ideal conditions. The calculation formula is ; where H(n) is the harmonic number, defined as: ; n is the number of samples in the tree; Anomaly Score(x) represents the anomaly score of sample x. The value range of the anomaly score is [0, 1]. The closer the score is to 1, the more likely the sample is an outlier. The anomaly score threshold is set according to business requirements and data distribution. Samples with anomaly scores greater than 0.9 are usually marked as high-risk threats.

[0041] Mark abnormal samples according to thresholds; record the causes of abnormalities; interpret the detection results and generate threat reports.

[0042] According to the numerical selection algorithm of the compliance coefficient, if the value is less than the system threshold, the cluster analysis algorithm is used. The steps are as follows: Step E1: Identify threat-related features and divide the data into training and test sets (e.g., 70% training set + 30% test set); ensure that the training set contains enough normal samples; Step E2: Select an appropriate clustering algorithm based on data characteristics and business requirements. If the data is spherically distributed, select K-means; if the data distribution is complex, select DBSCAN or OPTICS; Step E3: Initialize the model parameters according to the selected algorithm; example: for K-means, set the number of clusters; for DBSCAN, set epsilon and min_samples; Step E4: Use the training set data to train the clustering model; the model divides the data into multiple clusters by calculating the similarity between samples; The result of the clustering model is unlabeled clusters, each of which represents a group of behavior patterns with similar characteristics; example: the user behavior of a cluster is detected to be frequent access to sensitive resources, which may indicate unauthorized access behavior; Step E5: Identify abnormal behavior patterns by analyzing cluster characteristics; analyze the behavior characteristics of each cluster; identify clusters whose behavior patterns significantly deviate from normal patterns; example: detect that the user access frequency of one cluster is much higher than that of other clusters, which may indicate high-frequency attack behavior; interpret the detection results and generate a threat report.

[0043] In step S4, threat reports are received, a comprehensive risk report is generated, and the results are presented in a visual way, collating data from different detection methods to ensure that the report content covers all identified threats and their characteristics; lists the type of each threat, and explains the systems, departments, or business processes that each threat may affect; and provides specific examples or cases to support the description.

[0044] Use charts or dashboards to visually display risk distribution and trends. Examples: Line chart: displays the changing trend of access anomaly frequency; Bar chart: compares the quantitative distribution of different threat types; Heat map: displays the risk level distribution in different time periods.

[0045] Save the data generated by the system operation to facilitate staff verification.

[0046] Example 2 The present invention provides a security risk assessment system for digital archives, such as Figure 2 As shown, it mainly includes the following modules: Screening module, weighting module, and evaluation module; The screening module is used to conduct a preliminary analysis of the file features, preliminarily determine the weight of abnormal features, and count the amount of abnormal feature data; The weighting module weights different abnormal features. The weight of each abnormal feature is obtained by combining the entropy method and the CRITIC method. The evaluation module is used to further analyze the abnormal characteristics and determine the specific threat situation.

[0047] The evaluation module includes a selection module, which is used to select an isolation forest module or a cluster analysis module to further analyze the abnormal features according to the abnormal feature weight, timeliness, and abnormal feature data volume.

[0048] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0049] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0050] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0051] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0052] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A security risk assessment method for digital archives, characterized in that: The following steps are involved: Conduct a preliminary analysis of the archive features, preliminarily determine the weights of abnormal features, and organize the abnormal features; weight different abnormal features, and each abnormal feature weight is obtained by combining the entropy method and the CRITIC method; Determine the weight, timeliness, and amount of abnormal feature data of the abnormal features, calculate the compliance coefficient by weighted summation, and compare the compliance coefficient with the threshold to determine whether to use the isolation forest algorithm or cluster analysis algorithm to further analyze the threat-related features.

2. A security risk assessment method for digital archives according to claim 1, characterized in that: Organize the abnormal features, including cleaning the abnormal features and counting the amount of abnormal feature data.

3. A security risk assessment method for digital archives according to claim 1, characterized in that: The different abnormal features are weighted. Specifically, for the abnormal features, the difference coefficient is first calculated using the entropy method, and then the correlation is calculated using the CRITIC method; the weight of the abnormal feature is calculated using the weighted sum according to the difference coefficient and the correlation.

4. A security risk assessment method for digital archives according to claim 1, characterized in that: After a preliminary analysis of the archival features and screening out the abnormal features, the weight, timeliness, and amount of abnormal feature data are determined, and the compliance coefficient is calculated through weighted summation.

5. A security risk assessment method for digital archives according to claim 4, characterized in that: Determine the value of the compliance coefficient. If the value is greater than the system threshold, use the isolation forest algorithm to further analyze the abnormal features. If the value is less than the system threshold, select the cluster analysis algorithm to further analyze the abnormal features.

6. A security risk assessment system for digital archives, used to implement a security risk assessment method for digital archives as claimed in any one of claims 1 to 5, characterized in that: The evaluation system comprises: The screening module is used to conduct a preliminary analysis of the file features, preliminarily determine the weight of abnormal features, and count the amount of abnormal feature data; The weighting module weights different abnormal features. The weight of each abnormal feature is obtained by combining the entropy method and the CRITIC method. The evaluation module is used to further analyze the abnormal characteristics and determine the specific threat situation.

7. A security risk assessment system for digital archives according to claim 6, characterized in that: The evaluation module contains a selection module, which is used to select an isolation forest module or a cluster analysis module to further analyze the abnormal features according to the abnormal feature weight, timeliness, and abnormal feature data volume.

Images