Online data security compliance supervision method, system and medium for big data

Abstract

The present invention provides a method, system and medium for online data security compliance supervision for big data, which belongs to the field of data security supervision technology, including: obtaining data points that generate data anomalies in real-time online data streams, extracting the data segments corresponding to the real-time online data streams where the data nodes are located; calling a data compliance standard configuration file, and based on the data compliance standards, performing compliance judgments on the data segments for data loss, data anomalies and data tampering, and determining non-compliant data nodes; traversing the non-compliant data nodes, and determining the clustering density of a preset number of non-compliant data nodes on the time axis in each fixed time period; when the clustering density exceeds a preset threshold, the continuous time period between the time points where the first and last non-compliant data nodes are located is defined as the attacked time period. This method can quickly locate the key time period that is attacked and generates non-compliant data, and then perform data screening on multiple data streams, delete non-compliant data or correct data.

Description

Technical Field

[0001] The present invention relates to the technical field of data security compliance supervision, and in particular to a method, system and medium for online data security compliance supervision for big data. Background Art

[0002] Some of an enterprise's business-related data involves information privacy, trade secrets, audit tracking or intellectual property rights, and therefore requires data security compliance supervision. For example, manufacturing enterprises' production information, sales flow and other data, financial enterprises' transaction records, customer identity information and other data, medical institutions' storage of patient electronic medical records, health monitoring data and other data, e-commerce enterprises' user behavior logs, payment information and other data all require real-time compliance judgment and supervision of data loss, data anomalies and data tampering.

[0003] Traditional data security compliance monitoring relies on comparing current data against pre-set compliance standards to identify deviations or violations. However, in a big data environment, the exponential growth of data volumes and the highly dynamic nature of transmission paths make segment-by-segment data comparison impractical. To address this challenge, existing technologies typically employ sampling strategies, randomly selecting a subset of data nodes or focusing on specific nodes experiencing significant changes in data transmission status (such as peak traffic periods or cross-regional transmission nodes) for compliance analysis.

[0004] However, this random sampling approach can result in the omission of critical data nodes, and the selection of nodes with special statuses is susceptible to subjective judgment, making it difficult to fully cover potential risk points. In summary, in the context of large data volumes, the limitation of focusing only on the compliance of special nodes with significant changes not only weakens the objectivity and effectiveness of compliance supervision, but also may create hidden dangers to data security. Summary of the Invention

[0005] To solve the above problems, the present invention provides an online data security compliance supervision method for big data. The method can quickly locate the key time period that is attacked and generates non-compliant data, and then perform data screening on multiple data streams, delete non-compliant data or correct the data.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] An online data security compliance supervision system for big data includes the following steps:

[0008] Monitor multiple online data streams in real time through sliding time windows, obtain abnormal data nodes that generate data anomalies in each real-time online data stream, and extract the data segments corresponding to the real-time online data streams where the abnormal data nodes are located;

[0009] Call the data compliance standard configuration file and, based on the data compliance standard, perform compliance judgment on the data segment for data loss, data anomaly, and data tampering. Determine the non-compliant data features that produce loss features, anomaly features, and tampering features in the data segment. Identify the data nodes in the real-time online data stream that exhibit non-compliant data features as non-compliant data nodes.

[0010] Traverse the non-compliant data nodes and determine the clustering density of a preset number of non-compliant data nodes on the time axis in each fixed time period; when the clustering density exceeds a preset threshold, the continuous time period between the time points where the first and last non-compliant data nodes are located is defined as the attack time period;

[0011] Retrieve all data content corresponding to the attack time period of multiple real-time online data streams for compliance inspection to determine the cause of non-compliant data.

[0012] Preferably, the real-time monitoring of multiple online data streams through a sliding time window and obtaining abnormal data nodes generating data anomalies in each real-time online data stream comprises the following steps:

[0013] Receive online data streams in real time, set a sliding window of fixed time size, and slide the data within the window in real time;

[0014] Perform real-time analysis on the data in the sliding window, calculate the statistical information of the data, and determine whether there are data points exceeding the threshold value in the current sliding window based on the preset abnormal judgment threshold;

[0015] Perform threshold-based outlier detection on the data in the sliding window to obtain abnormal data points;

[0016] Data points that exceed the threshold and abnormal data points are regarded as abnormal data nodes.

[0017] Preferably, extracting the data segment corresponding to the real-time online data stream where the abnormal data node is located includes the following steps:

[0018] Mark the location of abnormal data nodes using timestamps in the data stream;

[0019] According to the size of the sliding window and the timestamp of the abnormal data node, the continuous data of n time points before and after the abnormal data point is extracted from the data stream to form the data segment to be checked for compliance.

[0020] Preferably, the compliance judgment of data loss, data anomaly and data tampering on the data segment includes the following steps:

[0021] Loading compliance standards related to data flows from configuration management systems;

[0022] Based on the loss criteria defined in the configuration file, determine whether there are missing data nodes in the data segment and obtain the data loss compliance judgment result;

[0023] Based on the abnormal threshold in the configuration file, determine whether there are data nodes in the data segment that exceed the normal range, and obtain the data abnormal compliance judgment result;

[0024] Perform matching verification based on the hash value set in the configuration file to determine whether there is a tampered data node in the data segment and obtain the data tampering compliance judgment result;

[0025] Non-compliant data nodes that cause data loss, data anomalies, and data tampering are labeled with types, and non-compliant data nodes and their corresponding information in all data streams are stored; the corresponding information of the non-compliant data nodes includes the corresponding timestamp, data value, and the data stream ID to which they belong.

[0026] Preferably, traversing the non-compliant data nodes and determining the aggregation density of a preset number of non-compliant data nodes on the time axis in each fixed time period includes the following steps:

[0027] Traverse all real-time data streams, mark each non-compliant data node, and store the non-compliant data node and its corresponding information;

[0028] Build a list of all non-compliant data nodes based on the non-compliant data nodes and their corresponding information, along with the timestamps of the corresponding non-compliant data nodes;

[0029] The preset fixed time period divides the data into multiple fixed-length time periods based on the timestamps in the data stream, starting at each non-compliant data node;

[0030] For each time period, count the number of non-compliant data nodes within that time period;

[0031] Determine the clustering density of non-compliant data nodes based on the number of non-compliant data nodes in each time period and the length of the time period:

[0032] Aggregation density = number of non-compliant data nodes / length of time period.

[0033] Preferably, when the aggregation density exceeds a preset threshold, the continuous time period between the time points where the first and last non-compliant data nodes are located is defined as the attacked time period, including the following steps:

[0034] For each time period, determine whether its aggregation density exceeds the preset threshold. If it exceeds the threshold, mark the time period as a time period with an attack;

[0035] Extract the timestamps of the first and last non-compliant data nodes in the time period where the attack occurred;

[0036] Based on the timestamps of the non-compliant data nodes at the beginning and end of each attack time period, the time interval between the attack time periods is determined. If the interval between the attack time periods is less than or equal to the preset maximum duration, they are considered as a continuous attack time period. If the interval between the attack time periods exceeds the preset time period, they are considered as multiple independent attack events.

[0037] According to the timestamps of the first and last non-compliant data nodes of the attack event, the total duration of the attacked time period is obtained, and the attack information is marked and stored; the attack information includes the timestamp, duration, and the data stream involved.

[0038] Preferably, the step of retrieving all data contents corresponding to the time period of the attack on multiple real-time online data streams for compliance checking and determining the cause of the non-compliant data includes the following steps:

[0039] Retrieve all real-time data streams related to the attack period, perform cross-checks, obtain abnormal status of multiple data streams, and analyze whether the abnormalities are caused by external factors.

[0040] Combine network monitoring, device logs, and external data to trace back the attack period and analyze whether there is a network attack, hardware failure, or transmission error.

[0041] Correct or delete the abnormal status of multiple data flows during the attack period.

[0042] The present invention also provides an online data security compliance supervision system for big data, the system comprising:

[0043] processor;

[0044] a memory having stored thereon a computer program executable on the processor;

[0045] Among them, when the computer program is executed by the processor, the steps of the online data security compliance supervision method for big data are implemented.

[0046] The present invention also provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, the steps of the online data security compliance supervision method for big data are implemented.

[0047] Beneficial effects of the present invention:

[0048] The present invention proposes an online data security compliance supervision method for big data, which quickly identifies data points that generate anomalies through a sliding time window, and judges whether non-compliant data is generated around the data points, including compliance judgments of data loss, data anomalies and data tampering, and further determines the characteristics of non-compliant data, and uses this as a means to quickly locate non-compliant data nodes and extract the time period when non-compliant data is generated. Based on the characteristics of non-compliant data aggregation, this method determines whether it has been attacked from the outside according to the aggregation density of non-compliant data nodes, and locates the time when the attack was made, and analyzes the data compliance of other data streams in this time period, thereby effectively improving the detection rate of non-compliant data where data anomalies are not obvious. This method can quickly determine data segments with a higher probability of non-compliance, use data segments where non-compliant feature data nodes are concentrated as key data segments, locate key time periods that are prone to non-compliant data flows, and investigate the causes of non-compliant data generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0050] Figure 2 This is a flow chart of a method for judging concentrated attacks based on cluster density according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Example 1

[0053] The traditional model for data security compliance supervision relies on comparing current data against pre-set compliance standards item by item to identify deviations or violations. However, in a big data environment, the scale of data is growing exponentially, and the transmission paths are highly dynamic, making segment-by-segment data comparison an unfeasible task. To address this challenge, regulators or companies typically adopt a sampling strategy, randomly selecting some data nodes or focusing on special nodes where data transmission status changes significantly (such as traffic peak periods and cross-regional transmission nodes) for compliance analysis. Although this method can improve efficiency, its inherent flaws cannot be ignored: random sampling may lead to the omission of key data nodes; the selection of nodes in special status is easily influenced by subjective judgment, making it difficult to fully cover potential risk points.

[0054] When an enterprise is receiving business data streams and is attacked or tampered with by external forces, non-compliant data transmission will be concentrated in a short period of time, or the data will return to compliant data within a certain range during subsequent transmissions due to the end of the attack or the correction of defense measures. However, for compliance supervision of big data, data compliance comparison cannot be performed one by one based on time nodes. However, when data streams are attacked by external forces, a large amount of non-compliant data will appear, that is, the characteristics of clustering of non-compliant data points will appear. Based on the characteristics of clustering of non-compliant data points, it is necessary to quickly determine the data segments with a higher probability of non-compliance, and use the data segments with concentrated non-compliant feature data nodes as key data segments, locate the key time periods that are prone to non-compliant data streams, and investigate the causes of non-compliant data generation. To this end, this embodiment proposes an online data security compliance supervision method for big data, and the specific steps are as follows: Figure 1 Shown, including:

[0055] S1: Monitor multiple online data streams in real time through sliding time windows, and obtain abnormal data nodes that generate data anomalies in each real-time online data stream.

[0056] S2: Extract the data segment corresponding to the real-time online data stream where the abnormal data node is located.

[0057] S3: Call the data compliance standard configuration file, and based on the data compliance standard, perform compliance judgment on the data segment for data loss, data anomaly, and data tampering, and determine the non-compliant data features that produce loss features, anomaly features, and tampering features in the data segment.

[0058] S4: Data nodes that have non-compliant data features in real-time online data streams are classified as non-compliant data nodes.

[0059] S5: Traverse the non-compliant data nodes to determine the aggregation density of a preset number of non-compliant data nodes on the time axis in each fixed time period.

[0060] S6: When the aggregation density exceeds a preset threshold, the continuous time period between the time points of the first and last non-compliant data nodes is defined as the attacked time period.

[0061] S7: Retrieve all data content corresponding to the attack period of multiple real-time online data streams for compliance inspection to determine the cause of non-compliant data.

[0062] In S1, the present invention first performs real-time monitoring of outliers and obtains abnormal data points through threshold judgment and data point outlier analysis, specifically including:

[0063] S1.1: Receive online data streams in real time, set a sliding window of fixed time size, and slide the data within the window in real time.

[0064] S1.2: Analyze the data in the sliding window in real time, calculate the statistical information of the data, and determine whether there are data points in the current sliding window that exceed the threshold based on the preset abnormal judgment threshold.

[0065] S1.3: Perform threshold-based outlier detection on the data in the sliding window to obtain abnormal data points.

[0066] S1.4: Data points exceeding the threshold and abnormal data points are treated as abnormal data nodes, and abnormal and exceeding threshold data points in the data window are marked in real time to prepare for subsequent compliance judgment.

[0067] Furthermore, in S2, the abnormal data node is used as a benchmark, and the continuous data before and after it is used as the minimum unit for compliance checking. Specifically, the location of the abnormal data node is marked with the timestamp in the data stream. Based on the size of the sliding window and the timestamp of the abnormal data node, the continuous data of n time points before and after the abnormal data point is extracted from the data stream to form the data segment to be checked for compliance.

[0068] The present invention provides the non-compliant abnormal forms generated after data is attacked, including loss, abnormality and tampering, and specifically includes the following steps:

[0069] S3.1: Load compliance standards associated with the data flow from the configuration management system.

[0070] S3.2: Based on the loss criteria defined in the configuration file, determine whether there are missing data nodes in the data segment and obtain a data loss compliance judgment result.

[0071] S3.3: Based on the abnormal threshold in the configuration file, determine whether there are data nodes in the data segment that exceed the normal range, and obtain the data abnormality compliance judgment result.

[0072] S3.4: Perform a matching check based on the hash value set in the configuration file to determine whether there is a tampered data node in the data segment and obtain a data tampering compliance judgment result.

[0073] S3.5: Label the types of non-compliant data nodes that cause data loss, data anomalies, and data tampering, and store the non-compliant data nodes and their corresponding information in all data streams; the corresponding information of the non-compliant data nodes includes the corresponding timestamp, data value, and the data stream ID to which they belong.

[0074] Specific, abnormal forms of non-compliance include:

[0075] (1) Data Loss

[0076] Data loss generally refers to the failure of data that should have existed to be recorded, stored, or transmitted successfully due to some reasons. This may result in missing, omitted, or incomplete data, especially when problems arise during data transmission, storage, or backup.

[0077] Data loss is characterized by missing fields or records, missing time series data, and discontinuous records. For example, if certain fields or records in a dataset are missing, this may result in empty values (NULL, NaN, etc.) appearing during data analysis, or the absence of certain items that should exist in the dataset. Alternatively, there may be no data records within a certain time period, making it impossible to analyze certain time points. Alternatively, if data is stored in chronological order, missing data often manifests as interruptions or jumps in the time series. For example, in an IoT device, if sensor data is lost, the temperature record at a certain point in time may be found to be null or NaN. In a database, if a data record fails to be submitted, the record may be missing.

[0078] Existing techniques for detecting data loss typically involve null value checks and time series checks. Null value checks detect null values (NaN, NULL) in data, such as using the isna() or isnull() functions in Pandas. Time series checks compare data from adjacent time points for missing values, using timestamp sorting to check the continuity of time series data.

[0079] (2) Data Anomalies

[0080] Data anomalies usually refer to the situation where, under certain conditions, data presents values or trends that are different from the majority of data points. This anomaly may be caused by system failure, sensor error, external environmental interference, etc.

[0081] Data anomalies can manifest as unusual fluctuations in values, extreme values, sudden changes in trends, and excessive fluctuations. For example, data may exhibit significant fluctuations outside the normal range, such as sudden jumps in sensor readings or the appearance of unreasonable values within a short period of time. Alternatively, certain data points may clearly exceed the normal range, far exceeding the range of historical data. Alternatively, data trends may abruptly change, such as long-term stable temperature data suddenly increasing or decreasing sharply at a specific point in time. Furthermore, unusually large fluctuations in data over a specific time period may indicate equipment failure or external interference.

[0082] Existing technologies typically use the standard deviation method, the IQR method, and time series anomaly detection for judgment. The standard deviation method calculates the mean and standard deviation of the data to identify outliers that differ by more than a certain number of standard deviations from the mean. The IQR method uses quartile analysis to identify data points outside the upper and lower quartiles. Time series anomaly detection uses time series analysis methods (such as ARIMA models and LSTM neural networks) to detect trend changes.

[0083] (3) Data Tampering

[0084] Data tampering refers to the unauthorized modification of data during storage, transmission or processing. The tampered data may cause data distortion and affect the accuracy and credibility of the data.

[0085] Data tampering can manifest as data inconsistency, changes in historical records, unexpected checksum or numeric changes, and suspicious timestamp modifications. For example, if the same data is inconsistent at different points in time or across different systems, data tampering or incorrect modification may have occurred. For example, transaction data may contain inconsistent amounts across different records. Modification or deletion of historical data may cause the data to be inconsistent with the actual events that occurred. Another example is a change in the data's hash value or checksum, which may indicate that the data was tampered with during transmission. Another example is a modified timestamp, where the recorded time does not match the actual time. Existing technologies typically use hash checksums.

[0086] Based on the characteristics of non-compliant data point aggregation, this paper provides a technical means for judging concentrated attacks by aggregation density. The flow chart is as follows: Figure 2 As shown, specifically including:

[0087] S5.1: Traverse all real-time data streams, mark each non-compliant data node, and store the non-compliant data node and its corresponding information.

[0088] S5.2: Construct a list of all non-compliant data nodes based on the non-compliant data nodes and their corresponding information, and the timestamps of the corresponding non-compliant data nodes.

[0089] S5.3: Preset fixed time periods: Based on the timestamps in the data stream, the data is divided into multiple fixed-length time periods, starting at each non-compliant data node. For example, if the timestamps in the data stream are from t1 to tn, the time periods can be defined as [t1, t1+5], [t1+5, t1+10], in minutes, and so on.

[0090] S5.4: For each time period, count the number of non-compliant data nodes in that time period. For example, for the time period [t1, t1+5], count the number of non-compliant data nodes that appear in that time period.

[0091] S5.5: Determine the cluster density of non-compliant data nodes based on the number of non-compliant data nodes in each time period and the length of the time period: Cluster density = number of non-compliant data nodes / length of time period. For example, if a time period is 5 minutes and there are 10 non-compliant data nodes in that time period, the cluster density is 2.

[0092] When the cluster density exceeds the preset threshold, the continuous time period between the time points of the first and last non-compliant data nodes is considered the attack time period. Specifically, the following steps are included:

[0093] S6.1: For each time period, determine whether its aggregation density exceeds a preset threshold. If it exceeds the threshold, mark the time period as a time period where an attack occurs.

[0094] S6.2: Extract the timestamps of the first and last non-compliant data nodes in the time period where the attack occurred, such as the time period [t_start, t_end], which represents the initial time range of the attacked time period.

[0095] S6.3: Determine the time interval between attack time periods based on the timestamps of the non-compliant data nodes at the beginning and end of each attack time period. If the time interval between attack time periods is less than or equal to the preset maximum duration, they are considered a single continuous attack time period. If the time interval between attack time periods exceeds the preset duration, they are considered multiple independent attack events.

[0096] S6.4: Based on the timestamps of the first and last non-compliant data nodes of the attack event, obtain the total duration of the attacked time period, mark and store the attack information; the attack information includes the timestamp, duration, and the data streams involved.

[0097] Furthermore, all data content corresponding to the attack period of multiple real-time online data streams is retrieved for compliance inspection to determine the cause of non-compliant data:

[0098] Retrieve all real-time data streams related to the attack period and perform a cross-check to identify multiple data stream anomalies. Analyze whether these anomalies are caused by external factors. Combine network monitoring, device logs, and external data to trace back to the attack period and analyze whether there are network attacks, hardware failures, or transmission errors. Correct or delete the multiple data stream anomalies during the attack period.

[0099] In this embodiment, this method utilizes a sliding time window technique to quickly identify data points that generate anomalies and, by analyzing the area surrounding the data points, determine whether non-compliant data has been generated. Specifically, this method not only determines whether data is lost, abnormal, or tampered with, but also conducts detailed compliance assessments for each type of non-compliant data, further identifying its non-compliant characteristics. This process helps quickly locate non-compliant data nodes and extract the time periods during which non-compliant data was generated. These time periods are considered potential anomaly intervals, providing precise location for subsequent analysis.

[0100] On this basis, the method pays special attention to the clustering characteristics of non-compliant data, and determines whether there are signs of external attacks by analyzing the clustering density of non-compliant data nodes. Specifically, by calculating the clustering density of non-compliant data in each time period, when the density exceeds the set threshold, the system can identify potential attack behaviors and further determine the time point when the attack occurs. This provides effective support for further analysis of the compliance of other data streams in this time period, especially when data anomalies are difficult to detect, greatly improving the detection rate of non-compliant data. With this technology, the system can not only accurately identify non-compliant data nodes, but also conduct in-depth investigations into the causes of non-compliant data.

[0101] Based on the clustering characteristics of non-compliant data, this method makes the identification of abnormal data more efficient, especially when faced with data streams where the anomalies are not obvious, and can extract problematic data segments through density analysis. Specifically, the method determines which time periods of data streams may have a higher risk of non-compliance based on the clustering density of non-compliant data nodes. These time periods are considered critical data segments, and the system will pay special attention to the data compliance during these time periods and deeply explore the root causes that may lead to data non-compliance. This monitoring method based on time period division effectively improves the detection capabilities of non-compliant data, can promptly identify and respond to external attacks, and thus ensure the security and compliance of data streams.

[0102] In this way, the system can quickly identify potential issues in data streams and pinpoint the time when data anomalies are most likely to occur. This approach not only provides an efficient solution for real-time data monitoring but also offers a strong guarantee for data security, particularly in scenarios where unusual behavior in data streams can be difficult to detect. By continuously monitoring and analyzing data stream compliance, the system can proactively identify and issue warnings about potential attacks, proactively preventing data security risks and providing strong support for subsequent processing and response measures.

[0103] The above is an online data security compliance supervision method for big data provided by one embodiment of this embodiment. Based on the same idea, this embodiment also provides a corresponding online data security compliance supervision system for big data. For the specific definition of the online data security compliance supervision system for big data, please refer to the definition of the online data security compliance supervision method for big data above, which will not be repeated here. The various modules in the above-mentioned online data security compliance supervision system for big data can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0104] This embodiment also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides an online data security compliance supervision method for big data.

[0105] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0106] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for online data security compliance supervision for big data, characterized by: The following steps are involved: Monitor multiple online data streams in real time through sliding time windows, obtain abnormal data nodes that generate data anomalies in each real-time online data stream, and extract the data segments corresponding to the real-time online data streams where the abnormal data nodes are located; Call the data compliance standard configuration file and, based on the data compliance standard, perform compliance judgment on the data segment for data loss, data anomaly, and data tampering. Determine the non-compliant data features that produce loss features, anomaly features, and tampering features in the data segment. Identify the data nodes in the real-time online data stream that exhibit non-compliant data features as non-compliant data nodes. Traverse all real-time data streams, mark each non-compliant data node, and store the non-compliant data node and its corresponding information; Build a list of all non-compliant data nodes based on the non-compliant data nodes and their corresponding information, along with the timestamps of the corresponding non-compliant data nodes; Based on the timestamps in the data stream, the data is divided into multiple fixed-length time periods with each non-compliant data node as the starting time point. For each time period, the number of non-compliant data nodes within that time period is counted. Based on the number of non-compliant data nodes in each time period and the length of the time period, the clustering density of non-compliant data nodes is determined: clustering density = number of non-compliant data nodes / length of the time period. For each time period, determine whether its cluster density exceeds the preset threshold. If it does, mark the time period as an attack time period. Extract the timestamps of the first and last non-compliant data nodes in the attack time period. Based on the timestamps of the non-compliant data nodes at the beginning and end of each attack time period, the time interval of the attack time period is determined; if the interval between the attack time periods is less than or equal to the preset maximum duration, it is considered as a continuous attack time period; if the interval between the attack time periods exceeds the preset time period, it is considered as multiple independent attack events; based on the timestamps of the non-compliant data nodes at the beginning and end of the attack event, the total duration of the attack time period is obtained, and the attack information is marked and stored; the attack information includes the timestamp, duration, and the data stream involved; Retrieve all data content corresponding to the attack time period of multiple real-time online data streams for compliance inspection to determine the cause of non-compliant data.

2. The online data security compliance supervision method for big data according to claim 1 is characterized in that: The method of monitoring multiple online data streams in real time through a sliding time window and obtaining abnormal data nodes generating data anomalies in each real-time online data stream includes the following steps: Receive online data streams in real time, set a sliding window of fixed time size, and slide the data within the window in real time; Perform real-time analysis on the data in the sliding window, calculate the statistical information of the data, and determine whether there are data points exceeding the threshold value in the current sliding window based on the preset abnormal judgment threshold; Perform threshold-based outlier detection on the data in the sliding window to obtain abnormal data points; Data points that exceed the threshold and abnormal data points are regarded as abnormal data nodes.

3. The online data security compliance supervision method for big data according to claim 2 is characterized in that: The step of extracting the data segment corresponding to the real-time online data stream where the abnormal data node is located includes the following steps: Mark the location of abnormal data nodes using timestamps in the data stream; According to the size of the sliding window and the timestamp of the abnormal data node, the continuous data of n time points before and after the abnormal data point is extracted from the data stream to form the data segment to be checked for compliance.

4. The online data security compliance supervision method for big data according to claim 3 is characterized in that: The compliance judgment of data loss, data anomaly and data tampering on the data segment includes the following steps: Loading compliance standards related to data flows from configuration management systems; Based on the loss criteria defined in the configuration file, determine whether there are missing data nodes in the data segment and obtain the data loss compliance judgment result; Based on the abnormal threshold in the configuration file, determine whether there are data nodes in the data segment that exceed the normal range, and obtain the data abnormal compliance judgment result; Perform matching verification based on the hash value set in the configuration file to determine whether there is a tampered data node in the data segment and obtain the data tampering compliance judgment result; Non-compliant data nodes that cause data loss, data anomalies, and data tampering are labeled with types, and non-compliant data nodes and their corresponding information in all data streams are stored; the corresponding information of the non-compliant data nodes includes the corresponding timestamp, data value, and the data stream ID to which they belong.

5. The online data security compliance supervision method for big data according to claim 1 is characterized in that: The step of retrieving all data contents corresponding to the time period of the attack on multiple real-time online data streams for compliance checking and determining the cause of the non-compliant data includes the following steps: Retrieve all real-time data streams related to the attack period, perform cross-checks, obtain abnormal status of multiple data streams, and analyze whether the abnormalities are caused by external factors. Combine network monitoring, device logs, and external data to trace back the attack period and analyze whether there is a network attack, hardware failure, or transmission error. Correct or delete the abnormal status of multiple data flows during the attack period.

6. An online data security compliance supervision system for big data, characterized by: The system comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the online data security compliance supervision method for big data as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the online data security compliance supervision method for big data as described in any one of claims 1 to 5.

Images