Data flow safety supervision system and method based on big data platform

Abstract

The invention discloses a data flow safety supervision system and method based on a big data platform, and relates to the technical field of safety supervision and data analysis, and the method comprises the following steps: collecting transmission process data and user operation data; the transmission process data comprises network flow data and encrypted information data; the user operation data comprises user behavior data, operation frequency and authority information data; analyzing the collected data, extracting risk factors, and carrying out standardization processing on risk factor data; constructing a dynamic data flow model by taking the preprocessed risk factor data as a base; analyzing potential risks in data flow, and constructing a comprehensive risk scoring model through deep learning; setting a risk threshold value; and according to the set risk threshold, dividing risk levels, and performing management according to the risk levels. According to the invention, through dynamic analysis of user behaviors and potential risks in a data transmission process, accurate risk assessment and intelligent level-to-level management are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety supervision and data analysis, in particular to a data flow safety supervision system and method based on a big data platform. BACKGROUND

[0002] The safety supervision field mainly supervises and controls safety, including following principles such as independence and openness, and involves many supervision subjects and objects. The data analysis technical field covers data collection, cleaning, storage, mining and visualization. In the aspect of fusion of the two, data can help safety supervision decision-making, realize real-time monitoring and early warning, risk assessment management, safety performance evaluation and accident investigation and analysis, thereby improving the safety supervision level.

[0003] With the rapid development of big data technology, data flow plays a crucial role in the operation of enterprises and organizations. However, the complexity and diversity of data flow pose challenges to safety supervision, it is difficult to detect abnormal behavior in data transmission in real time; it is difficult to quickly analyze and respond to safety problems in real-time data flow; traditional risk monitoring methods mainly rely on rule bases, which are difficult to adapt to emerging attack behaviors and abnormal operations, and the risk classification is relatively fuzzy, resulting in inefficient resource allocation and management strategies. SUMMARY

[0004] The purpose of the present application is to provide a data flow safety supervision system and method based on a big data platform to solve the problems in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a data flow safety supervision method based on a big data platform, comprising:

[0006] Collecting transmission process data and user operation data;

[0007] The transmission process data includes network traffic data and encryption information data;

[0008] The user operation data includes user behavior data, operation frequency and permission information data;

[0009] Analyzing the collected data, extracting risk factors, and standardizing the risk factor data;

[0010] Using the pre-processed risk factor data as a base, a dynamic data flow model is constructed;

[0011] Analyzing potential risks in data flow, and constructing a comprehensive risk scoring model through deep learning;

[0012] Setting a first risk threshold;

[0013] Setting a second risk threshold;

[0014] According to the set threshold, the risk level is divided, and management is carried out according to the risk level.

[0015] Collect transmission process data and user operation data, the transmission process data includes network traffic data and encryption information data; the user operation data includes user behavior data, operation frequency and permission information data; specifically:

[0016] The network traffic data includes the size, frequency, destination IP, transmission protocol and transmission delay of the transmission data;

[0017] The encryption information data includes the encryption algorithm and encryption key used in the transmission process;

[0018] Monitor the data packets, traffic and network connections in the transmission process through network traffic analysis tools;

[0019] The network traffic analysis tools include Wireshark, NetFlow, Zeek, etc.

[0020] The transmission protocol includes HTTP, FTP, etc.

[0021] Detect and record abnormal traffic and attack behavior in the network through the access intrusion detection system IDS and intrusion prevention system IPS;

[0022] For encrypted transmission data, decrypted traffic is needed to check for unauthorized access or other abnormal behavior, using SSL or TLS analysis;

[0023] Record the requests, responses and user behavior in the transmission process through the proxy server;

[0024] The proxy server includes Nginx, Squid, etc.

[0025] The user behavior data includes user reading, modifying, deleting, copying and downloading operations on data;

[0026] The user behavior data also includes whether to use privileged accounts, access sensitive data, etc.

[0027] Monitor user behavior in data usage process in real time through Exabeam and Securonix tools;

[0028] Get the user's operation frequency and user's permission information data.

[0029] Including whether there is abnormal frequent access, batch operation, etc.

[0030] File operation logs are recorded using file monitoring tools AuditD and OSSEC to ensure that user operation frequency and user permission information are recorded.

[0031] The collected data is analyzed to extract risk factors, and the risk factor data is then standardized, specifically as follows:

[0032] Risk factors were extracted using principal component analysis (PCA).

[0033] Among these methods, extracting risk factors can also involve historical data analysis and summarization, as well as expert advice.

[0034] The risk factors are denoted as {x1, x2, ..., x...} n}; where x n Let x represent the nth risk factor, where n represents the total number of risk factors and is a positive integer; i ∈{x1,x2,…,x n}, x i Let i represent the i-th risk factor, where i represents the type label of different risk factors, and i is a positive integer, 1≤i≤n;

[0035] Standardization is performed to eliminate the influence of different factor units and magnitudes.

[0036] The standardization process includes Z-score standardization and Min-Max standardization.

[0037] The standardized processing method was selected based on historical experience.

[0038] The Z-score standardization specifically refers to:

[0039] Among them, z i This represents the standardized data; x i μ represents the i-th risk factor, where i represents the type label of different risk factors; i σi represents the mean of the i-th risk factor; σi represents the standard deviation of the i-th risk factor.

[0040] The Min-Max standardization specifically refers to:

[0041] Among them, z i This represents the standardized data; x i Let represent the i-th risk factor, where i represents the type label of different risk factors; max(x i ) and min(x i ) represent the maximum and minimum values ​​of the i-th risk factor, respectively.

[0042] Based on the pre-processed data, a dynamic data flow model is constructed, specifically:

[0043] The dynamic data flow model includes constructing a full life cycle graph, behavior flow analysis, and edge weight analysis.

[0044] A full life cycle graph is constructed from data generation to destruction. The nodes in the graph represent each flow stage, and the edges represent the data flow relationship between different stages.

[0045] Each node represents a data flow stage, including data generation, data transmission, data storage, data usage, data sharing, and data destruction.

[0046] Among them, each node can be:

[0047] Data generation: the initial stage of data generation, involving data sources and generation methods.

[0048] Data transmission: data is transmitted from one location to another (such as network transmission, file transmission, etc.);

[0049] Data storage: the saving stage of data in databases, file systems, or cloud storage.

[0050] Data usage: the process of reading, analyzing, or modifying data.

[0051] Data sharing: the process of sharing data with other departments or external organizations.

[0052] Data destruction: the process of final destruction or complete deletion of data to ensure irrecoverability.

[0053] These nodes clearly define each flow stage of the data life cycle, allowing for the establishment of corresponding security monitoring measures for each stage.

[0054] Among them, the edges represent the data flow relationship between different stages, which can be:

[0055] Generation to transmission: data flows from the creation stage to the transmission stage.

[0056] Transmission to storage: data enters the storage stage after transmission is complete.

[0057] Storage to usage: data is accessed and used at a certain point in time in the storage stage.

[0058] Usage to sharing: data is shared by multiple departments or external organizations.

[0059] Sharing to destruction: shared data is destroyed after use is completed.

[0060] The behavior flow analysis is based on historical transmission process data and user operation data to analyze the differences between normal and abnormal behaviors, and to obtain common behavior patterns of users, devices and systems at different data stages; the user behavior flow analysis is dynamically performed and continuously adjusted and updated as time goes by and behavior changes.

[0061] User behavior patterns may change over time, depending on their job content or access permissions, so a dynamic adjustment mechanism is needed to ensure that the model can continuously reflect the current reality.

[0062] Among them, the usage patterns, operation frequency, access time and location information of historical transmission process data and user operation data are extracted, and time, environment and frequency factors are combined as auxiliary factors;

[0063] The normal behavior refers to the data operations performed by users and devices in the system according to predetermined rules and business needs; for example, a user may access a specific data table daily to query or update it.

[0064] The abnormal behavior refers to data operations that are not performed in accordance with predetermined rules and business requirements; for example, frequent access to sensitive data, large-scale data export, and login from different locations.

[0065] in,

[0066] Time factor: When the behavior occurred, and whether there was any unusual activity outside of working hours;

[0067] Environmental factors: the physical environment and network environment of the user or device (such as VPN, IP address, geographical location);

[0068] Frequency factor: The frequency of operations, frequent requests, or large-scale data access may be flagged as abnormal;

[0069] By incorporating these factors into the model, abnormal behavior can be identified more accurately.

[0070] Based on factors such as the time, frequency, environment, and visitor identity of data flow, the risk coefficient of each behavior is calculated using the entropy weight method, and the risk coefficient of the behavior is used as the edge weight of the dynamic data flow model.

[0071] The risk coefficient for each behavior is calculated using the entropy weight method, specifically as follows:

[0072] The entropy weighting method can reflect the information content of each factor by calculating its entropy value, thereby assigning a reasonable weight to each factor.

[0073] Select m sets of data as samples from historical transmission process data and user operation data, where m is a positive integer;

[0074] A decision matrix Z is constructed for each risk factor and each sample, z ij ∈Z, wherein z ij represents the value of the i-th risk factor of the j-th sample after standardization, j represents the sample label, j is a positive integer, 1≤j≤m;

[0075] Calculate the information entropy H

[0076]

[0077] wherein H i represents the information entropy of the i-th risk factor; k represents the entropy change constant, k=1 / lnm;

[0078] Calculate the risk coefficient w

[0079]

[0080] wherein w i represents the risk coefficient of the i-th risk factor.

[0081] Analyze the potential risks in data flow, and build a comprehensive risk score model through deep learning, specifically:

[0082] Based on the dynamic data flow model, analyze the potential risks in data flow, extract and classify through SVM, and build a comprehensive risk score model;

[0083] The comprehensive risk score model is specifically:

[0084]

[0085] wherein R represents the comprehensive risk score; f i (z i ) represents the fitting mapping function of the i-th risk factor; z i represents the standardized data; β represents the adjustment factor, β∈[0,1];

[0086] wherein the adjustment factor β is used to control the strength of the nonlinear effect, the greater the adjustment factor, the stronger the nonlinear response of the model to the factor.

[0087] wherein the function mapping is realized through the nonlinear mapping module in MATLAB and the pandas library in Python.

[0088] Specifically:

[0089] Set the first risk threshold Q1;

[0090] Set the second risk threshold Q2;

[0091] Equipped with an early warning device;

[0092] When the comprehensive risk score R is greater than or equal to the first risk threshold Q1, it is classified as high risk;

[0093] When the comprehensive risk score R is less than the first risk threshold Q1 but greater than or equal to the second risk threshold Q2, it is classified as medium risk;

[0094] When the comprehensive risk score R is less than the second risk threshold Q2, it is classified as low risk;

[0095] According to the classified risk level, the pre-prepared management strategy is used for early warning and management.

[0096] A data flow security supervision system based on a big data platform, comprising a data collection module, a feature selection module, a dynamic data flow module, a risk assessment module, and an early warning and management module;

[0097] The data collection module is used to collect transmission process data and user operation data, wherein the transmission process data includes network traffic data and encryption information data; and the user operation data includes user behavior data, operation frequency, and permission information data;

[0098] The feature selection module is used to analyze the collected data, extract, and standardize risk factor data;

[0099] The dynamic data flow module is used to construct a data full-life cycle graph and dynamically analyze user behavior flow;

[0100] The risk assessment module is used to construct a comprehensive risk score model based on deep learning and perform real-time risk assessment;

[0101] The early warning and management module is used to set risk thresholds, divide risk levels according to the risk thresholds, and implement corresponding early warning and management strategies for different risk levels.

[0102] The data collection module comprises a software adaptation unit and a data collection unit;

[0103] The software adaptation unit is used to install and test data collection software;

[0104] The data collection software includes network traffic analysis tools, intrusion detection systems (IDS), and intrusion prevention systems (IPS), etc.;

[0105] The data collection unit is used to collect transmission process data and user operation data.

[0106] The early warning and management module provides a user interaction interface, supports visual display of real-time comprehensive risk scores, and displays historical management strategies, thereby providing a reference for management personnel.

[0107] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention constructs a dynamic data flow model, combines principal component analysis, entropy weight method and deep learning technology to cover the security supervision of the entire data lifecycle, dynamically analyzes user behavior and potential risks in the data transmission process, achieves accurate risk assessment and intelligent hierarchical management, has real-time early warning capabilities, strong adaptability and versatility, effectively overcomes the shortcomings of traditional methods in terms of real-time performance, accuracy and intelligence, and provides comprehensive support for data security. Attached Figure Description

[0108] Fig. 1 This is a schematic diagram of the structure of a data flow security supervision method based on a big data platform according to the present invention;

[0109] Fig. 2 This is a schematic diagram of a data flow security monitoring system based on a big data platform according to the present invention. Detailed Implementation

[0110] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0111] Example: Figs. 1-2 As shown, the present invention provides a technical solution, such as... Fig. 1 As shown, a data flow security supervision method based on a big data platform includes:

[0112] Collect and transmit data and user operation data;

[0113] The data transmitted includes network traffic data and encrypted information data;

[0114] The user operation data includes user behavior data, operation frequency, and permission information data;

[0115] Analyze the collected data, extract risk factors, and standardize the risk factor data;

[0116] Based on the preprocessed risk factor data, a dynamic data flow model is constructed;

[0117] Analyze potential risks in data flow and build a comprehensive risk scoring model using deep learning;

[0118] Set a first risk threshold;

[0119] A second risk threshold is set;

[0120] According to the set threshold, the risk level is divided, and management is carried out according to the risk level.

[0121] Collect transmission process data and user operation data, the transmission process data includes network traffic data and encryption information data; The user operation data includes user behavior data, operation frequency and permission information data; Specifically:

[0122] The network traffic data includes the size, frequency, destination IP, transmission protocol and transmission delay of the transmission data;

[0123] The encryption information data includes the encryption algorithm and encryption key used in the transmission process;

[0124] Through the network traffic analysis tool, the data packet, traffic and network connection in the transmission process are monitored;

[0125] Among them, the network traffic analysis tool includes Wireshark, NetFlow, Zeek, etc.;

[0126] Among them, the transmission protocol includes HTTP, FTP, etc.;

[0127] Through the access of intrusion detection system IDS and intrusion prevention system IPS, abnormal traffic and attack behavior in the network are detected and recorded;

[0128] Among them, for the encrypted transmission data, the decrypted traffic needs to be checked for unauthorized access or other abnormal behavior, using SSL or TLS analysis;

[0129] Through the proxy server, the request, response and user behavior in the transmission process are recorded;

[0130] The proxy server includes Nginx, Squid, etc.;

[0131] The user behavior data includes user reading, modifying, deleting, copying and downloading operations on data;

[0132] Among them, the user behavior data also includes whether to use privileged accounts, whether to access sensitive data, etc.

[0133] Through Exabeam and Securonix tools, the user's behavior in the data use process is monitored in real time;

[0134] Get the user's operation frequency and user's permission information data.

[0135] Among them, including whether there is abnormal frequent access, batch operation, etc.

[0136] The file operation log is recorded by AuditD and OSSEC, and the operation frequency of the user and the permission information of the user are recorded.

[0137] The collected data is analyzed, and risk factors are extracted, and the risk factor data is standardized, specifically:

[0138] The risk factors are extracted by principal component analysis (PCA) method;

[0139] The risk factors can also be extracted by historical data analysis and expert recommendations, etc.

[0140] The risk factors are denoted as {x1, x2, …, x n}; wherein x n represents the nth risk factor, n represents the total number of risk factors, n is a positive integer; x i ∈{x1, x2, …, x n}, x i represents the ith risk factor, i represents the type label of different risk factors, i is a positive integer, and 1≤i≤n;

[0141] The standardization processing is performed to eliminate the influence of different factor units and magnitudes.

[0142] The standardization processing includes Z-score standardization and Min-Max standardization.

[0143] The standardization processing method is selected according to historical experience;

[0144] The Z-score standardization is specifically:

[0145] wherein z i represents the standardized data; x i represents the ith risk factor, i represents the type label of different risk factors; μ i represents the average value of the ith risk factor; σi represents the standard deviation of the ith risk factor.

[0146] The Min-Max standardization is specifically:

[0147] wherein z i represents the standardized data; x i represents the ith risk factor, i represents the type label of different risk factors; max(x i ) and min(x i ) represent the maximum and minimum values of the ith risk factor, respectively.

[0148] Based on the pre-processed data, a dynamic data flow model is constructed, specifically:

[0149] The dynamic data flow model includes constructing a full life cycle graph, behavior flow analysis, and edge weight analysis;

[0150] A full life cycle graph is constructed from data generation to destruction. The nodes in the graph represent each flow stage. The edges represent the data flow relationship between different stages.

[0151] Each node represents a data flow stage, including data generation, data transmission, data storage, data usage, data sharing, and data destruction.

[0152] Each node can be:

[0153] Data generation: the initial stage of data generation, involving data sources and generation methods.

[0154] Data transmission: data is transmitted from one place to another (such as network transmission, file transmission, etc.);

[0155] Data storage: the stage of saving data in databases, file systems, or cloud storage;

[0156] Data usage: the process of reading, analyzing, or modifying data;

[0157] Data sharing: the process of sharing data with other departments or external organizations;

[0158] Data destruction: the process of final destruction or complete deletion of data to ensure irrecoverability;

[0159] These nodes clearly define each flow stage of the data life cycle, allowing for the establishment of corresponding security monitoring measures for each stage.

[0160] The edges represent the data flow relationship between different stages and can be:

[0161] Generation to transmission: data flows from the creation stage to the transmission stage;

[0162] Transmission to storage: data enters the storage stage after transmission is complete;

[0163] Storage to usage: data is accessed and used at a certain point in time in the storage stage;

[0164] Usage to sharing: data is shared by multiple departments or external organizations;

[0165] Sharing to destruction: shared data is destroyed after use is complete.

[0166] The behavior flow analysis is based on historical transmission process data and user operation data to analyze the differences between normal behavior and abnormal behavior, and obtain the common behavior patterns of users, devices and systems at different data stages; dynamic user behavior flow analysis is performed, and the model is continuously adjusted and updated as time goes on and behavior changes;

[0167] The behavior patterns of users may change over time, work content or access permissions, so a dynamic adjustment mechanism is needed to ensure that the model can continuously reflect the current reality.

[0168] Among them, the usage patterns, operation frequency, access time and location information of historical transmission process data and user operation data are extracted, and time, environment, frequency factors and other auxiliary factors are combined;

[0169] The normal behavior is the data operation of users and devices in the system according to the predetermined rules and business requirements; for example, a user accesses a specific data table for query or update every day.

[0170] The abnormal behavior is the behavior of data operation that does not follow the predetermined rules and business requirements; for example, frequent access to sensitive data, large-scale data export, and off-site login.

[0171] Among them,

[0172] Time factor: the time when the behavior occurs, whether there is abnormal activity outside working hours;

[0173] Environmental factor: the physical environment and network environment (such as VPN, IP address, geographic location) where the user or device is located;

[0174] Frequency factor: the frequency of operation, frequent requests or large-scale data access may be marked as abnormal;

[0175] By combining these factors into the model, abnormal behavior can be more accurately identified.

[0176] Based on the time, frequency, environment and visitor identity factors of data flow, the risk coefficient of each behavior is calculated by entropy weight method, and the risk coefficient of behavior is taken as the edge weight of dynamic data flow model.

[0177] The risk coefficient of each behavior is calculated by entropy weight method, which is:

[0178] Entropy weight method can reflect the information amount of each factor by calculating the entropy value of each factor, so as to assign a reasonable weight to each factor.

[0179] Select m groups of data from historical transmission process data and user operation data as samples, m is a positive integer;

[0180] A decision matrix Z is constructed for each risk factor and each sample, z ij ∈Z, where z ij represents the value of the i-th risk factor of the j-th sample after standardization, j represents the sample label, j is a positive integer, 1≤j≤m;

[0181] Calculate the information entropy:

[0182]

[0183] where H i represents the information entropy of the i-th risk factor; k represents the entropy change constant, k = 1 / lnm;

[0184] Calculate the risk coefficient:

[0185]

[0186] where w i represents the risk coefficient of the i-th risk factor.

[0187] Analyze the potential risks in data flow, and build a comprehensive risk score model through deep learning, specifically:

[0188] Based on the dynamic data flow model, analyze the potential risks in data flow, extract and classify through SVM, and build a comprehensive risk score model;

[0189] The comprehensive risk score model is specifically:

[0190]

[0191] where R represents the comprehensive risk score; f i (z i ) represents the fitting mapping function of the i-th risk factor; z i represents the standardized data; β represents the adjustment factor, β ∈ [0, 1];

[0192] where the adjustment factor β is used to control the strength of the nonlinear effect, the greater the adjustment factor, the stronger the nonlinear response of the model to the factor.

[0193] where the function mapping is realized through the nonlinear mapping module in MATLAB and the pandas library in Python.

[0194] where, for the operation frequency:

[0195] The operation frequency represents the access frequency of a user or system to data, if the operation frequency is too high, there may be risks of abuse of authority, data leakage or malicious behavior;

[0196] The function form of the mapping is fitted by the pandas library of Python:

[0197] f1(z1) = ln(1+z1);

[0198] For data transmission with a normalized value z1=0.8 for a certain operation frequency, the local operation frequency risk score is f1(0.8)=ln(1+0.8)≈0.5878;

[0199] wherein, for data sensitivity:

[0200] Data sensitivity refers to the sensitivity level of the data itself, for example, personal information, financial data and other highly sensitive information may cause higher security risks;

[0201] The function form of the mapping is fitted by the nonlinear mapping module in MATLAB:

[0202] f2(z2)=e z2 -1;

[0203] For data transmission with a normalized value z2=0.7 for data sensitivity, the local data sensitivity risk score is f2(0.7)=e 0.7 -1≈1.403; (higher sensitivity, risk significantly increased).

[0204] wherein, for data transmission delay:

[0205] Data transmission delay refers to the transmission time of data from one node to another node. Longer delay may indicate network bottlenecks, malicious interference or transmission channel abnormalities.

[0206] The function form of the mapping is fitted by the nonlinear mapping module in MATLAB:

[0207] f3(z3)=(z3) 1 / 2 ;

[0208] For data transmission with a normalized value z3=0.4 for data transmission delay, the local data sensitivity risk score is f3(0.4)=(0.4) 1 / 2 ≈0.6325.

[0209] Specifically:

[0210] Set the first risk threshold Q1;

[0211] Set the second risk threshold Q2;

[0212] Equipped with an early warning device;

[0213] When the comprehensive risk score R is greater than or equal to the first risk threshold Q1, it is classified as high risk;

[0214] When the comprehensive risk score R is less than the first risk threshold Q1 but greater than or equal to the second risk threshold Q2, it is classified as medium risk;

[0215] When the comprehensive risk score R is less than the second risk threshold Q2, it is classified as low risk;

[0216] According to the classified risk level, the pre-prepared management strategy is used for early warning and management.

[0217] Among them, for high risk:

[0218] Trigger high-risk warning and alarm through multiple channels: SMS, phone notification, system pop-up window, email notification, etc. to inform the relevant responsible person and security team for real-time response.

[0219] Automatic emergency response:

[0220] Block risk operation: immediately suspend data flow or limit user access.

[0221] Isolate risk area: isolate abnormal behavior or data in a controlled environment (sandbox running or isolated network).

[0222] Force login verification: re-perform multi-factor authentication (MFA) for related operations.

[0223] Detailed record and traceability:

[0224] Collect detailed operation logs, network traffic, behavior patterns, etc.

[0225] Perform real-time traceability, analyze the source of abnormal behavior, and confirm whether the threat is caused by internal or external.

[0226] Further review and repair:

[0227] Investigate the root cause of the problem and confirm whether there are system vulnerabilities, configuration errors or malicious operations.

[0228] Repair potential problems, update security policies and improve system protection capabilities.

[0229] Business impact assessment:

[0230] Confirm the potential impact of the risk on business and customers, and develop follow-up remediation measures.

[0231] For medium risk:

[0232] Trigger medium-risk warning and notify data administrators or security personnel to pay attention to related behavior through email or system notification.

[0233] Dynamic Permission Adjustment:

[0234] Temporary Limitation of Operational Permissions: Restrict sensitive data operations for relevant users or processes.

[0235] Access Frequency Monitoring: Strengthen monitoring of abnormal access frequencies.

[0236] Behavior Analysis and Confirmation:

[0237] Mark relevant behaviors as under review for further analysis to confirm whether they are false positives.

[0238] Combine historical data and behavior patterns to assess whether to upgrade to high risk or downgrade to low risk.

[0239] Risk Mitigation Measures:

[0240] Suggest users change passwords or re-perform security verification.

[0241] Remind relevant users or departments to follow data use security regulations.

[0242] Regular Tracking and Reporting:

[0243] Track medium-risk behaviors for a period of time to confirm whether the risk persists or disappears.

[0244] Record risks in system logs for subsequent analysis and modeling optimization.

[0245] For Low Risk:

[0246] Record risk behaviors in the system background, but do not trigger early warnings;

[0247] Classify and archive risk behaviors, marked as "under observation";

[0248] User Education and Specification Reminders:

[0249] Send non-mandatory reminders to relevant users, such as attention to access frequency or specification operation behavior.

[0250] Provide security guidelines to help users understand the correct data operation process.

[0251] Behavior Pattern Update:

[0252] Record the behavior in the behavior flow model and adjust the baseline of normal behavior.

[0253] If the behavior frequently occurs and does not cause actual threats, it can be gradually classified as normal behavior.

[0254] Daily Monitoring:

[0255] Maintain monitoring of low-risk behaviors;

[0256] If low-risk behaviors frequently co-occur with other abnormal behaviors, consider upgrading to medium risk.

[0257] Optimization algorithm and model:

[0258] Optimize existing risk assessment algorithms using data from low-risk behaviors to reduce false positive and false negative rates.

[0259] As shown in Fig. 2 A data flow security supervision system based on a big data platform includes a data collection module, a feature selection module, a dynamic data flow module, a risk assessment module, and an early warning and management module.

[0260] The data collection module is used to collect transmission process data and user operation data. The transmission process data includes network traffic data and encryption information data. The user operation data includes user behavior data, operation frequency, and permission information data.

[0261] The feature selection module is used to analyze the collected data, extract, and standardize risk factor data.

[0262] The dynamic data flow module is used to construct a data full-life cycle graph and dynamically analyze user behavior flow.

[0263] The risk assessment module is used to construct a comprehensive risk scoring model based on deep learning for real-time risk assessment.

[0264] The early warning and management module is used to set risk thresholds, divide risk levels based on risk thresholds, and implement corresponding early warning and management strategies for different risk levels.

[0265] The data collection module includes a software adaptation unit and a data collection unit.

[0266] The software adaptation unit is used to install and test data collection software.

[0267] The data collection software includes network traffic analysis tools, intrusion detection systems (IDS), and intrusion prevention systems (IPS).

[0268] The data collection unit is used to collect transmission process data and user operation data.

[0269] The early warning and management module provides a user interaction interface, supports visual display of real-time comprehensive risk scores, and displays historical management strategies, providing a reference for management personnel.

[0270] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. A data flow security supervision method based on a big data platform, characterized in that: include: Collect and transmit data and user operation data; The data transmitted includes network traffic data and encrypted information data; The user operation data includes user behavior data, operation frequency, and permission information data; Analyze the collected data, extract risk factors, and standardize the risk factor data; Based on the preprocessed risk factor data, a dynamic data flow model is constructed; Analyze potential risks in data flow and build a comprehensive risk scoring model using deep learning; Set a first risk threshold; Set a second risk threshold; Based on the set risk thresholds, risk levels are classified and managed accordingly.

2. The data flow security supervision method based on a big data platform according to claim 1, characterized in that: The data collected during the transmission process and user operation data are specifically as follows: The network traffic data includes the size, frequency, destination IP, transmission protocol, and transmission delay of the transmitted data; The encrypted information data includes the encryption algorithm and encryption key used during transmission; Monitor data packets, traffic, and network connectivity during transmission using network traffic analysis tools; By connecting to an Intrusion Detection System (IDS) and an Intrusion Prevention System (IPS), abnormal traffic and attack behaviors in the network are detected and recorded. The proxy server records requests, responses, and user behavior during the transmission process. The user behavior data includes user operations such as reading, modifying, deleting, copying, and downloading data; Obtain user operation frequency and user permission information data.

3. The data flow security supervision method based on a big data platform according to claim 2, characterized in that: The collected data is analyzed to extract risk factors, and the risk factor data is then standardized, specifically as follows: Risk factors were extracted using principal component analysis (PCA). The risk factors are denoted as {x1, x2, ..., x...} n }; where x n Let x represent the nth risk factor, where n represents the total number of risk factors and is a positive integer; i ∈{x1,x2,…,x n }, x i Let i represent the i-th risk factor, where i represents the type label of different risk factors, and i is a positive integer, 1≤i≤n; The standardization process includes Z-score standardization and Min-Max standardization.

4. The data flow security supervision method based on a big data platform according to claim 3, characterized in that: Based on the preprocessed data, a dynamic data flow model is constructed, specifically as follows: The dynamic data flow model includes constructing a full lifecycle map, behavioral flow analysis, and edge weight analysis. Construct a full lifecycle map of data from generation to destruction; the nodes in the map represent various stages of the data flow. Edges represent the data flow relationships between different stages; Each node represents a stage of data flow, including: data generation, data transmission, data storage, data usage, data sharing, and data destruction. The behavior flow analysis is based on historical transmission process data and user operation data to analyze the differences between normal and abnormal behaviors, and to obtain common behavior patterns of users, devices and systems at different data stages; and to dynamically perform user behavior flow analysis. The normal behavior refers to the data operations performed by users and devices in the system according to predetermined rules and business requirements. The abnormal behavior refers to data operations that are not performed in accordance with predetermined rules and business requirements. Based on factors such as the time, frequency, environment, and visitor identity of data flow, the risk coefficient of each behavior is calculated using the entropy weight method, and the risk coefficient of the behavior is used as the edge weight of the dynamic data flow model.

5. The data flow security supervision method based on a big data platform according to claim 4, characterized in that: The risk coefficient for each behavior is calculated using the entropy weight method, specifically as follows: Select m sets of data as samples from historical transmission process data and user operation data, where m is a positive integer; Constructing the decision matrix: For each risk factor and each sample, construct the decision matrix Z, z ij ∈Z, where z ij Let represent the standardized value of the i-th risk factor of the j-th sample, where j represents the sample label and is a positive integer, 1 ≤ j ≤ m; Calculate information entropy: Among them, H i Let represent the information entropy of the i-th risk factor; k represents the entropy constant, k = 1 / lnm; Calculate the risk coefficient: Among them, w i This represents the risk coefficient of the i-th risk factor.

6. The data flow security supervision method based on a big data platform according to claim 5, characterized in that: To analyze potential risks in data flow, a comprehensive risk scoring model is constructed using deep learning, specifically as follows: Based on the dynamic data flow model, potential risks in data flow are analyzed, classified and extracted using SVM, and a comprehensive risk scoring model is constructed. The comprehensive risk scoring model is as follows: Where R represents the comprehensive risk score; f i (z i ) represents the fitted mapping function for the i-th risk factor; z i represents the standardized data; β represents the adjustment factor, β∈[0,1].

7. The data flow security supervision method based on a big data platform according to claim 6, characterized in that: Specifically: Set the first risk threshold Q1; Set a second risk threshold Q2; Equipped with early warning devices; When the comprehensive risk score R is greater than or equal to the first risk threshold Q1, it is classified as high risk; When the comprehensive risk score R is less than the first risk threshold Q1 but greater than or equal to the second risk threshold Q2, it is classified as medium risk; When the comprehensive risk score R is less than the second risk threshold Q2, it is classified as low risk; Based on the risk levels, early warning and management are carried out in accordance with pre-established management strategies.

8. A data flow security monitoring system based on a big data platform, employing the data flow security monitoring method based on a big data platform as described in any one of claims 1-7, characterized in that: The data flow security monitoring system includes a data acquisition module, a feature selection module, a dynamic data flow module, a risk assessment module, and an early warning and management module. The data acquisition module is used to collect transmission process data and user operation data. The transmission process data includes network traffic data and encrypted information data; the user operation data includes user behavior data, operation frequency, and permission information data. The feature selection module is used to analyze the collected data, extract and standardize risk factor data; The dynamic data flow module is used to construct a full lifecycle data map and dynamically analyze user behavior flow. The risk assessment module is used to build a comprehensive risk scoring model based on deep learning and to perform real-time risk assessment. The early warning and management module is used to set risk thresholds, classify risk levels based on risk thresholds, and implement corresponding early warning and management strategies for different risk levels.

9. A data flow security monitoring system based on a big data platform according to claim 8, characterized in that: The data acquisition module includes a software adaptation unit and a data acquisition unit; The software adaptation unit is used to install and test the data acquisition software; The data acquisition unit is used to collect transmission process data and user operation data.

10. A data flow security monitoring system based on a big data platform according to claim 9, characterized in that: The early warning and management module provides a user interaction interface, supports the visualization of real-time comprehensive risk scores, and displays historical management strategies for reference by managers.

Images