A data flow security supervision system and method based on a big data platform

Abstract

This invention discloses a data flow security monitoring system and method based on a big data platform, relating to the fields of security monitoring and data analysis technology. The system includes the following steps: collecting transmission process data and user operation data; the transmission process data includes network traffic data and encrypted information data; the user operation data includes user behavior data, operation frequency, and permission information data; analyzing the collected data, extracting risk factors, and standardizing the risk factor data; constructing a dynamic data flow model based on the pre-processed risk factor data; analyzing potential risks in data flow and constructing a comprehensive risk scoring model through deep learning; setting risk thresholds; classifying risk levels according to the set risk thresholds and managing them accordingly. This invention achieves accurate risk assessment and intelligent hierarchical management by dynamically analyzing user behavior and potential risks in the data transmission process.

Description

Technical Field

[0001] This invention relates to the field of security supervision and data analysis technology, specifically a data flow security supervision system and method based on a big data platform. Background Technology

[0002] The field of safety supervision primarily involves monitoring and regulating safety, adhering to principles such as independence and transparency, and encompassing numerous regulatory bodies and entities. The field of data analytics covers data collection, cleaning, storage, mining, and visualization. In their integration, data can support safety regulatory decision-making, enabling real-time monitoring and early warning, risk assessment and management, safety performance evaluation, and accident investigation and analysis, thereby improving the level of safety supervision.

[0003] With the rapid development of big data technology, data flow plays a crucial role in the operations of enterprises and organizations. However, the complexity and diversity of data flow pose challenges to security supervision. Abnormal behavior in data transmission is difficult to detect in real time; security issues in real-time data flow cannot be quickly analyzed and addressed; traditional risk monitoring methods mainly rely on rule bases, which are difficult to adapt to emerging attack behaviors and abnormal operations, and risk classification is relatively vague, resulting in inefficient resource allocation and management strategies. Summary of the Invention

[0004] The purpose of this invention is to provide a data flow security monitoring system and method based on a big data platform to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data flow security supervision method based on a big data platform, comprising:

[0006] Collect and transmit data and user operation data;

[0007] The data transmitted includes network traffic data and encrypted information data;

[0008] The user operation data includes user behavior data, operation frequency, and permission information data;

[0009] Analyze the collected data, extract risk factors, and standardize the risk factor data;

[0010] Based on the preprocessed risk factor data, a dynamic data flow model is constructed;

[0011] Analyze potential risks in data flow and build a comprehensive risk scoring model using deep learning;

[0012] Set a first risk threshold;

[0013] Set a second risk threshold;

[0014] Based on the set thresholds, risk levels are classified and managed accordingly.

[0015] The system collects transmission process data and user operation data. The transmission process data includes network traffic data and encrypted information data; the user operation data includes user behavior data, operation frequency, and permission information data. Specifically:

[0016] The network traffic data includes the size, frequency, destination IP, transmission protocol, and transmission delay of the transmitted data;

[0017] The encrypted information data includes the encryption algorithm and encryption key used during transmission;

[0018] Monitor data packets, traffic, and network connectivity during transmission using network traffic analysis tools;

[0019] The network traffic analysis tools mentioned include Wireshark, NetFlow, Zeek, etc.

[0020] The transmission protocols include HTTP, FTP, etc.

[0021] By connecting to an Intrusion Detection System (IDS) and an Intrusion Prevention System (IPS), abnormal traffic and attack behaviors in the network are detected and recorded.

[0022] For encrypted data transmission, the traffic needs to be decrypted to check for unauthorized access or other abnormal behavior, using SSL or TLS analysis.

[0023] The proxy server records requests, responses, and user behavior during the transmission process.

[0024] The proxy servers include Nginx, Squid, etc.

[0025] The user behavior data includes user operations such as reading, modifying, deleting, copying, and downloading data;

[0026] The user behavior data also includes whether privileged accounts are used and whether sensitive data is accessed.

[0027] Monitor user behavior in real time during data usage using Exabeam and Securonix tools;

[0028] Obtain user operation frequency and user permission information data.

[0029] This includes checking for abnormally frequent access or batch operations.

[0030] File operation logs are recorded using file monitoring tools AuditD and OSSEC to ensure that user operation frequency and user permission information are recorded.

[0031] The collected data is analyzed to extract risk factors, and the risk factor data is then standardized, specifically as follows:

[0032] Risk factors were extracted using principal component analysis (PCA).

[0033] Among these methods, extracting risk factors can also involve historical data analysis and summarization, as well as expert advice.

[0034] The risk factors are denoted as {x1, x2, ..., x}. n}; where x n Let x represent the nth risk factor, where n represents the total number of risk factors and is a positive integer; i ∈{x1,x2,…,x n}, x i Let i represent the i-th risk factor, where i represents the type label of different risk factors, and i is a positive integer, 1≤i≤n;

[0035] Standardization is performed to eliminate the influence of different factor units and magnitudes.

[0036] The standardization process includes Z-score standardization and Min-Max standardization.

[0037] The standardized processing method was selected based on historical experience.

[0038] The Z-score standardization specifically refers to:

[0039] Among them, z i This represents the standardized data; x i μ represents the i-th risk factor, where i represents the type label of different risk factors; i σi represents the mean of the i-th risk factor; σi represents the standard deviation of the i-th risk factor.

[0040] The Min-Max standardization specifically refers to:

[0041] Among them, z i This represents the standardized data; x i Let represent the i-th risk factor, where i represents the type label of different risk factors; max(x i ) and min(x i ) represent the maximum and minimum values ​​of the i-th risk factor, respectively.

[0042] Based on the preprocessed data, a dynamic data flow model is constructed, specifically as follows:

[0043] The dynamic data flow model includes constructing a full lifecycle map, behavioral flow analysis, and edge weight analysis.

[0044] Construct a full lifecycle graph of data from generation to destruction; the nodes in the graph represent various flow stages; the edges represent the data flow relationships between different stages;

[0045] Each node represents a stage of data flow, including: data generation, data transmission, data storage, data usage, data sharing, and data destruction.

[0046] Each node can be:

[0047] Data generation: The initial stage in which data is generated, involving data sources and generation methods;

[0048] Data transmission: The transfer of data from one location to another (such as network transmission, file transfer, etc.);

[0049] Data storage: The stage in which data is stored in a database, file system, or cloud storage;

[0050] Data usage: The process by which data is read, analyzed, or modified;

[0051] Data sharing: The process by which data is shared with other departments or external organizations;

[0052] Data destruction: The process of ultimately destroying or completely deleting data to ensure it is unrecoverable;

[0053] These nodes clearly define each stage of the data lifecycle, enabling the establishment of corresponding security monitoring measures for each stage.

[0054] Here, the edges represent the data flow relationships between different stages, which can be:

[0055] From creation to transmission: Data flows from the creation phase into the transmission phase;

[0056] Data is transferred to storage: After the transfer is complete, the data enters the storage stage;

[0057] Storage to Use: Data is accessed and used at a certain point in time during the storage phase;

[0058] Data sharing is used: Data is shared among multiple departments or external organizations;

[0059] Share to Destroy: Shared data is destroyed after it is no longer in use.

[0060] The behavior flow analysis is based on historical transmission process data and user operation data to analyze the differences between normal and abnormal behaviors, and to obtain common behavior patterns of users, devices and systems at different data stages; the user behavior flow analysis is dynamically performed and continuously adjusted and updated as time goes by and behavior changes.

[0061] User behavior patterns may change over time, depending on their job content or access permissions, so a dynamic adjustment mechanism is needed to ensure that the model can continuously reflect the current reality.

[0062] Among them, the usage patterns, operation frequency, access time and location information of historical transmission process data and user operation data are extracted, and time, environment and frequency factors are combined as auxiliary factors;

[0063] The normal behavior refers to the data operations performed by users and devices in the system according to predetermined rules and business needs; for example, a user may access a specific data table daily to query or update it.

[0064] The abnormal behavior refers to data operations that are not performed in accordance with predetermined rules and business requirements; for example, frequent access to sensitive data, large-scale data export, and login from different locations.

[0065] in,

[0066] Time factor: When the behavior occurred, and whether there was any unusual activity outside of working hours;

[0067] Environmental factors: the physical environment and network environment of the user or device (such as VPN, IP address, geographical location);

[0068] Frequency factor: The frequency of operations, frequent requests, or large-scale data access may be flagged as abnormal;

[0069] By incorporating these factors into the model, abnormal behavior can be identified more accurately.

[0070] Based on factors such as the time, frequency, environment, and visitor identity of data flow, the risk coefficient of each behavior is calculated using the entropy weight method, and the risk coefficient of the behavior is used as the edge weight of the dynamic data flow model.

[0071] The risk coefficient for each behavior is calculated using the entropy weight method, specifically as follows:

[0072] The entropy weighting method can reflect the information content of each factor by calculating its entropy value, thereby assigning a reasonable weight to each factor.

[0073] Select m sets of data as samples from historical transmission process data and user operation data, where m is a positive integer;

[0074] Constructing the decision matrix: For each risk factor and each sample, construct the decision matrix Z, z ij ∈Z, where z ij Let represent the standardized value of the i-th risk factor of the j-th sample, where j represents the sample label and is a positive integer, 1 ≤ j ≤ m;

[0075] Calculate information entropy:

[0076]

[0077] Among them, H i Let represent the information entropy of the i-th risk factor; k represents the entropy constant, k = 1 / lnm;

[0078] Calculate the risk coefficient:

[0079]

[0080] Among them, w i This represents the risk coefficient of the i-th risk factor.

[0081] To analyze potential risks in data flow, a comprehensive risk scoring model is constructed using deep learning, specifically as follows:

[0082] Based on the dynamic data flow model, potential risks in data flow are analyzed, classified and extracted using SVM, and a comprehensive risk scoring model is constructed.

[0083] The comprehensive risk scoring model is as follows:

[0084]

[0085] Where R represents the comprehensive risk score; f i (z i ) represents the fitted mapping function for the i-th risk factor; z i This represents the standardized data; β represents the adjustment factor, β∈[0,1];

[0086] The adjustment factor β is used to control the intensity of nonlinear effects. The larger the adjustment factor, the stronger the nonlinear response of the model to the factor.

[0087] The function mapping is implemented using the nonlinear mapping module in MATLAB and the pandas library in Python.

[0088] Specifically:

[0089] Set the first risk threshold Q1;

[0090] Set a second risk threshold Q2;

[0091] Equipped with early warning devices;

[0092] When the comprehensive risk score R is greater than or equal to the first risk threshold Q1, it is classified as high risk;

[0093] When the comprehensive risk score R is less than the first risk threshold Q1 but greater than or equal to the second risk threshold Q2, it is classified as medium risk;

[0094] When the comprehensive risk score R is less than the second risk threshold Q2, it is classified as low risk;

[0095] Based on the risk levels, early warning and management are carried out in accordance with pre-established management strategies.

[0096] A data flow security monitoring system based on a big data platform, the data flow security monitoring system includes a data acquisition module, a feature selection module, a dynamic data flow module, a risk assessment module, and an early warning and management module;

[0097] The data acquisition module is used to collect transmission process data and user operation data. The transmission process data includes network traffic data and encrypted information data; the user operation data includes user behavior data, operation frequency, and permission information data.

[0098] The feature selection module is used to analyze the collected data, extract and standardize risk factor data;

[0099] The dynamic data flow module is used to construct a full lifecycle data map and dynamically analyze user behavior flow.

[0100] The risk assessment module is used to build a comprehensive risk scoring model based on deep learning and to perform real-time risk assessment.

[0101] The early warning and management module is used to set risk thresholds, classify risk levels based on risk thresholds, and implement corresponding early warning and management strategies for different risk levels.

[0102] The data acquisition module includes a software adaptation unit and a data acquisition unit;

[0103] The software adaptation unit is used to install and test the data acquisition software;

[0104] The data acquisition software includes network traffic analysis tools, intrusion detection systems (IDS), and intrusion prevention systems (IPS), etc.

[0105] The data acquisition unit is used to collect transmission process data and user operation data.

[0106] The early warning and management module provides a user interaction interface, supports the visualization of real-time comprehensive risk scores, and displays historical management strategies for reference by managers.

[0107] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention constructs a dynamic data flow model, combines principal component analysis, entropy weight method and deep learning technology to cover the security supervision of the entire data lifecycle, dynamically analyzes user behavior and potential risks in the data transmission process, achieves accurate risk assessment and intelligent hierarchical management, has real-time early warning capabilities, strong adaptability and versatility, effectively overcomes the shortcomings of traditional methods in terms of real-time performance, accuracy and intelligence, and provides comprehensive support for data security. Attached Figure Description

[0108] Figure 1 This is a schematic diagram of the structure of a data flow security supervision method based on a big data platform according to the present invention;

[0109] Figure 2 This is a schematic diagram of a data flow security monitoring system based on a big data platform according to the present invention. Detailed Implementation

[0110] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0111] Example: Figures 1-2 As shown, the present invention provides a technical solution, such as... Figure 1 As shown, a data flow security supervision method based on a big data platform includes:

[0112] Collect and transmit data and user operation data;

[0113] The data transmitted includes network traffic data and encrypted information data;

[0114] The user operation data includes user behavior data, operation frequency, and permission information data;

[0115] Analyze the collected data, extract risk factors, and standardize the risk factor data;

[0116] Based on the preprocessed risk factor data, a dynamic data flow model is constructed;

[0117] Analyze potential risks in data flow and build a comprehensive risk scoring model using deep learning;

[0118] Set a first risk threshold;

[0119] Set a second risk threshold;

[0120] Based on the set thresholds, risk levels are classified and managed accordingly.

[0121] The system collects transmission process data and user operation data. The transmission process data includes network traffic data and encrypted information data; the user operation data includes user behavior data, operation frequency, and permission information data. Specifically:

[0122] The network traffic data includes the size, frequency, destination IP, transmission protocol, and transmission delay of the transmitted data;

[0123] The encrypted information data includes the encryption algorithm and encryption key used during transmission;

[0124] Monitor data packets, traffic, and network connectivity during transmission using network traffic analysis tools;

[0125] The network traffic analysis tools mentioned include Wireshark, NetFlow, Zeek, etc.

[0126] The transmission protocols include HTTP, FTP, etc.

[0127] By connecting to an Intrusion Detection System (IDS) and an Intrusion Prevention System (IPS), abnormal traffic and attack behaviors in the network are detected and recorded.

[0128] For encrypted data transmission, the traffic needs to be decrypted to check for unauthorized access or other abnormal behavior, using SSL or TLS analysis.

[0129] The proxy server records requests, responses, and user behavior during the transmission process.

[0130] The proxy servers include Nginx, Squid, etc.

[0131] The user behavior data includes user operations such as reading, modifying, deleting, copying, and downloading data;

[0132] The user behavior data also includes whether privileged accounts are used and whether sensitive data is accessed.

[0133] Monitor user behavior in real time during data usage using Exabeam and Securonix tools;

[0134] Obtain user operation frequency and user permission information data.

[0135] This includes checking for abnormally frequent access or batch operations.

[0136] File operation logs are recorded using file monitoring tools AuditD and OSSEC to ensure that user operation frequency and user permission information are recorded.

[0137] The collected data is analyzed to extract risk factors, and the risk factor data is then standardized, specifically as follows:

[0138] Risk factors were extracted using principal component analysis (PCA).

[0139] Among these methods, extracting risk factors can also involve historical data analysis and summarization, as well as expert advice.

[0140] The risk factors are denoted as {x1, x2, ..., x}. n}; where x n Let x represent the nth risk factor, where n represents the total number of risk factors and is a positive integer; i ∈{x1,x2,…,x n}, x i Let i represent the i-th risk factor, where i represents the type label of different risk factors, and i is a positive integer, 1≤i≤n;

[0141] Standardization is performed to eliminate the influence of different factor units and magnitudes.

[0142] The standardization process includes Z-score standardization and Min-Max standardization.

[0143] The standardized processing method was selected based on historical experience.

[0144] The Z-score standardization specifically refers to:

[0145] Among them, z i This represents the standardized data; x i μ represents the i-th risk factor, where i represents the type label of different risk factors; i σi represents the mean of the i-th risk factor; σi represents the standard deviation of the i-th risk factor.

[0146] The Min-Max standardization specifically refers to:

[0147] Among them, z i This represents the standardized data; x i Let represent the i-th risk factor, where i represents the type label of different risk factors; max(x i ) and min(x i ) represent the maximum and minimum values ​​of the i-th risk factor, respectively.

[0148] Based on the preprocessed data, a dynamic data flow model is constructed, specifically as follows:

[0149] The dynamic data flow model includes constructing a full lifecycle map, behavioral flow analysis, and edge weight analysis.

[0150] Construct a full lifecycle graph of data from generation to destruction; the nodes in the graph represent various flow stages; the edges represent the data flow relationships between different stages;

[0151] Each node represents a stage of data flow, including: data generation, data transmission, data storage, data usage, data sharing, and data destruction.

[0152] Each node can be:

[0153] Data generation: The initial stage in which data is generated, involving data sources and generation methods;

[0154] Data transmission: The transfer of data from one location to another (such as network transmission, file transfer, etc.);

[0155] Data storage: The stage in which data is stored in a database, file system, or cloud storage;

[0156] Data usage: The process by which data is read, analyzed, or modified;

[0157] Data sharing: The process by which data is shared with other departments or external organizations;

[0158] Data destruction: The process of ultimately destroying or completely deleting data to ensure it is unrecoverable;

[0159] These nodes clearly define each stage of the data lifecycle, enabling the establishment of corresponding security monitoring measures for each stage.

[0160] Here, the edges represent the data flow relationships between different stages, which can be:

[0161] From creation to transmission: Data flows from the creation phase into the transmission phase;

[0162] Data is transferred to storage: After the transfer is complete, the data enters the storage stage;

[0163] Storage to Use: Data is accessed and used at a certain point in time during the storage phase;

[0164] Data sharing is used: Data is shared among multiple departments or external organizations;

[0165] Share to Destroy: Shared data is destroyed after it is no longer in use.

[0166] The behavior flow analysis is based on historical transmission process data and user operation data to analyze the differences between normal and abnormal behaviors, and to obtain common behavior patterns of users, devices and systems at different data stages; the user behavior flow analysis is dynamically performed and continuously adjusted and updated as time goes by and behavior changes.

[0167] User behavior patterns may change over time, depending on their job content or access permissions, so a dynamic adjustment mechanism is needed to ensure that the model can continuously reflect the current reality.

[0168] Among them, the usage patterns, operation frequency, access time and location information of historical transmission process data and user operation data are extracted, and time, environment and frequency factors are combined as auxiliary factors;

[0169] The normal behavior refers to the data operations performed by users and devices in the system according to predetermined rules and business needs; for example, a user may access a specific data table daily to query or update it.

[0170] The abnormal behavior refers to data operations that are not performed in accordance with predetermined rules and business requirements; for example, frequent access to sensitive data, large-scale data export, and login from different locations.

[0171] in,

[0172] Time factor: When the behavior occurred, and whether there was any unusual activity outside of working hours;

[0173] Environmental factors: the physical environment and network environment of the user or device (such as VPN, IP address, geographical location);

[0174] Frequency factor: The frequency of operations, frequent requests, or large-scale data access may be flagged as abnormal;

[0175] By incorporating these factors into the model, abnormal behavior can be identified more accurately.

[0176] Based on factors such as the time, frequency, environment, and visitor identity of data flow, the risk coefficient of each behavior is calculated using the entropy weight method, and the risk coefficient of the behavior is used as the edge weight of the dynamic data flow model.

[0177] The risk coefficient for each behavior is calculated using the entropy weight method, specifically as follows:

[0178] The entropy weighting method can reflect the information content of each factor by calculating its entropy value, thereby assigning a reasonable weight to each factor.

[0179] Select m sets of data as samples from historical transmission process data and user operation data, where m is a positive integer;

[0180] Constructing the decision matrix: For each risk factor and each sample, construct the decision matrix Z, z ij ∈Z, where z ij Let represent the standardized value of the i-th risk factor of the j-th sample, where j represents the sample label and is a positive integer, 1 ≤ j ≤ m;

[0181] Calculate information entropy:

[0182]

[0183] Among them, H i Let represent the information entropy of the i-th risk factor; k represents the entropy constant, k = 1 / lnm;

[0184] Calculate the risk coefficient:

[0185]

[0186] Among them, w i This represents the risk coefficient of the i-th risk factor.

[0187] To analyze potential risks in data flow, a comprehensive risk scoring model is constructed using deep learning, specifically as follows:

[0188] Based on the dynamic data flow model, potential risks in data flow are analyzed, classified and extracted using SVM, and a comprehensive risk scoring model is constructed.

[0189] The comprehensive risk scoring model is as follows:

[0190]

[0191] Where R represents the comprehensive risk score; f i (z i ) represents the fitted mapping function for the i-th risk factor; z i This represents the standardized data; β represents the adjustment factor, β∈[0,1];

[0192] The adjustment factor β is used to control the intensity of nonlinear effects. The larger the adjustment factor, the stronger the nonlinear response of the model to the factor.

[0193] The function mapping is implemented using the nonlinear mapping module in MATLAB and the pandas library in Python.

[0194] Regarding the operating frequency:

[0195] Operation frequency refers to how often a user or system accesses data. If the operation frequency is too high, there may be risks of abuse of privileges, data leakage, or malicious behavior.

[0196] Fitting is performed using Python's pandas library, and the mapping function takes the following form:

[0197] f1(z1) = ln(1+z1);

[0198] For a data transmission with a normalized operation frequency of z1 = 0.8, the local operation frequency risk score is f1(0.8) = ln(1+0.8) ≈ 0.5878;

[0199] Regarding data sensitivity:

[0200] Data sensitivity refers to the level of sensitivity of the data itself. For example, highly sensitive information such as personal identification information and financial data may pose higher security risks.

[0201] Fitting is performed using the nonlinear mapping module in MATLAB. The mapping function has the following form:

[0202] f2(z2)=e z2 -1;

[0203] For data transmission with a standardized data sensitivity value z2 = 0.7, the local data sensitivity risk score is f2(0.7) = e 0.7 -1≈1.403; (Higher sensitivity, significantly increased risk).

[0204] Specifically, regarding data transmission latency:

[0205] Data transmission latency refers to the time it takes for data to travel from one node to another. Longer latency may indicate network bottlenecks, malicious interference, or transmission channel anomalies.

[0206] Fitting is performed using the nonlinear mapping module in MATLAB. The mapping function has the following form:

[0207] f3(z3)=(z3) 1 / 2 ;

[0208] For data transmission with a standardized latency of z3 = 0.4, the local data sensitivity risk score is f3(0.4) = (0.4). 1 / 2 ≈0.6325.

[0209] Specifically:

[0210] Set the first risk threshold Q1;

[0211] Set a second risk threshold Q2;

[0212] Equipped with early warning devices;

[0213] When the comprehensive risk score R is greater than or equal to the first risk threshold Q1, it is classified as high risk;

[0214] When the comprehensive risk score R is less than the first risk threshold Q1 but greater than or equal to the second risk threshold Q2, it is classified as medium risk;

[0215] When the comprehensive risk score R is less than the second risk threshold Q2, it is classified as low risk;

[0216] Based on the risk levels, early warning and management are carried out in accordance with pre-established management strategies.

[0217] Among them, for high-risk:

[0218] Trigger a high-risk warning and notify relevant personnel and the security team in real time through multiple channels, including SMS, telephone notification, system pop-ups, and email notifications.

[0219] Automated emergency response:

[0220] Block risky operations: Immediately suspend data flow or restrict user access.

[0221] Isolate high-risk areas: Isolate anomalous behavior or data in a controlled environment (sandbox operation or isolated network).

[0222] Forced login authentication: Re-perform multi-factor authentication (MFA) for relevant operations.

[0223] Detailed records and source tracing:

[0224] Collect detailed operation logs, network traffic, behavior patterns, and other information.

[0225] Conduct real-time source tracing to analyze the origin of abnormal behavior and confirm whether the threat is caused internally or externally.

[0226] Further review and fixes:

[0227] Investigate the root cause of the problem to determine if there are system vulnerabilities, configuration errors, or malicious operations.

[0228] Fix potential issues, update security policies, and improve system protection capabilities.

[0229] Business impact assessment:

[0230] Identify the potential impact of the risk on the business and customers, and develop subsequent remedial measures.

[0231] For medium risk:

[0232] If a medium-risk warning is triggered, the data administrator or security personnel will be notified via email or in-system notification to pay attention to the relevant behavior.

[0233] Dynamic permission adjustment:

[0234] Temporarily restrict operation permissions: Restrict sensitive data operations of relevant users or processes.

[0235] Access frequency monitoring: Enhance the monitoring of abnormal access frequencies.

[0236] Behavioral analysis and confirmation:

[0237] The relevant behaviors will be marked as pending review and further analysis will be conducted to confirm whether they are false alarms.

[0238] Based on historical data and behavioral patterns, assess whether it is necessary to upgrade to high risk or downgrade to low risk.

[0239] Risk mitigation measures:

[0240] Users are advised to change their passwords or re-verify their security credentials.

[0241] Remind relevant users or departments to comply with data security regulations.

[0242] Regular tracking and reporting:

[0243] Track medium-risk behaviors for a period of time to confirm whether the risk persists or disappears.

[0244] Risks are recorded in the system log for subsequent analysis and modeling optimization.

[0245] For low risk:

[0246] Record risky behaviors in the system background, but do not trigger alerts;

[0247] Classify and file risky behaviors, and mark them as "to be observed";

[0248] User education and guidelines reminders:

[0249] Send non-mandatory reminders to relevant users, such as asking them to pay attention to the frequency of access or to regulate their operating behavior.

[0250] Provide security guidelines to help users understand the correct data handling procedures.

[0251] Behavioral pattern update:

[0252] Record this behavior in the behavior flow model and adjust the baseline of normal behavior.

[0253] If this behavior occurs frequently and does not pose a real threat, it can be gradually classified as normal behavior.

[0254] Routine monitoring:

[0255] Maintain monitoring of low-risk behaviors;

[0256] If low-risk behaviors frequently occur together with other abnormal behaviors, consider upgrading to medium-risk.

[0257] Optimization algorithms and models:

[0258] Optimize existing risk assessment algorithms using data from low-risk behaviors to reduce false positives and false negatives.

[0259] like Figure 2 As shown, a data flow security monitoring system based on a big data platform is disclosed. The data flow security monitoring system includes a data acquisition module, a feature selection module, a dynamic data flow module, a risk assessment module, and an early warning and management module.

[0260] The data acquisition module is used to collect transmission process data and user operation data. The transmission process data includes network traffic data and encrypted information data; the user operation data includes user behavior data, operation frequency, and permission information data.

[0261] The feature selection module is used to analyze the collected data, extract and standardize risk factor data;

[0262] The dynamic data flow module is used to construct a full lifecycle data map and dynamically analyze user behavior flow.

[0263] The risk assessment module is used to build a comprehensive risk scoring model based on deep learning and to perform real-time risk assessment.

[0264] The early warning and management module is used to set risk thresholds, classify risk levels based on risk thresholds, and implement corresponding early warning and management strategies for different risk levels.

[0265] The data acquisition module includes a software adaptation unit and a data acquisition unit;

[0266] The software adaptation unit is used to install and test the data acquisition software;

[0267] The data acquisition software includes network traffic analysis tools, intrusion detection systems (IDS), and intrusion prevention systems (IPS), etc.

[0268] The data acquisition unit is used to collect transmission process data and user operation data.

[0269] The early warning and management module provides a user interaction interface, supports the visualization of real-time comprehensive risk scores, and displays historical management strategies for reference by managers.

[0270] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for supervising data flow security based on a big data platform, characterized in that: The application relates to a dynamic data flow risk management method and system. The method comprises the following steps: collecting transmission process data and user operation data; the transmission process data comprises network traffic data and encryption information data; the user operation data comprises user behavior data, operation frequency and permission information data; analyzing the collected data, extracting risk factors and standardizing the risk factor data; constructing a dynamic data flow model based on the preprocessed risk factor data; specifically: the dynamic data flow model comprises constructing a whole life cycle graph, behavior flow analysis and edge weight analysis; constructing a whole life cycle graph from data generation to data destruction; the nodes in the graph represent various flow stages; the edges represent the data flow relationship between different stages; each node represents a data flow stage, including data generation, data transmission, data storage, data use, data sharing and data destruction; based on the time, frequency, environment and visitor identity factors of data flow, the risk coefficient of each behavior is calculated by an entropy weight method, and the risk coefficient of the behavior is taken as the edge weight of the dynamic data flow model; specifically: Constructing decision matrix: for each risk factor and each sample, a decision matrix Z, z ij ∈ Z, where z ij represents the value of the i-th risk factor of the j-th sample after standardization, j represents the sample label, j is a positive integer, 1≤j≤m; Computing information entropy: ; wherein H i represents the information entropy of the i-th risk factor; k represents an entropy change constant, k = 1 / lnm; Computing the risk coefficient: ; wherein w i represents the risk coefficient of the i-th risk factor; selecting m groups of data from historical transmission process data and user operation data as samples, wherein m is a positive integer; analyzing potential risks in data flow and constructing a comprehensive risk scoring model through deep learning; setting a first risk threshold; setting a second risk threshold; 2.The data flow security supervision method based on a big data platform according to claim 1, characterized in that: according to the set risk threshold, dividing the risk level and managing according to the risk level. The collected transmission process data and user operation data are as follows: the network traffic data comprises the size, frequency, destination IP, transmission protocol and transmission delay of the transmission data; the encryption information data comprises encryption algorithms and encryption keys used in the transmission process; monitoring data packets, traffic and network connections in the transmission process through a network traffic analysis tool; detecting and recording abnormal traffic and attack behaviors in the network through an intrusion detection system (IDS) and an intrusion prevention system (IPS); recording requests, responses and user behaviors in the transmission process through a proxy server; the user behavior data comprises user reading, modifying, deleting, copying and downloading operations on data; 3.The data flow security supervision method based on a big data platform of claim 2, wherein: obtaining the operation frequency of the user and the permission information data of the user. The collected data is analyzed, risk factors are extracted and the risk factor data is standardized, specifically: The risk factors are denoted as {x1, x2, ..., x}. n }; where x n Let x represent the nth risk factor, where n represents the total number of risk factors and is a positive integer; i ∈{x1,x2,…,x n }, x i Let i represent the i-th risk factor, where i represents the type label of different risk factors, and i is a positive integer, 1≤i≤n; risk factors are extracted through principal component analysis (PCA) method; 4. The data flow security supervision method based on the big data platform according to claim 3, characterized in that: the standardization process comprises Z-score standardization and Min-Max standardization. The preprocessed data is taken as a basis to construct a dynamic data flow model, which further comprises: the behavior flow analysis is based on historical transmission process data and user operation data, and differences between normal behaviors and abnormal behaviors are analyzed to obtain common behavior patterns of users, devices and systems in different data stages; dynamic user behavior flow analysis is performed; the normal behavior is a data operation performed by a user and a device in the system according to a predetermined rule and business demand; 5. The data flow security supervision method based on the big data platform according to claim 4, characterized in that: the abnormal behavior is a behavior that does not perform data operation according to the predetermined rule and business demand. The potential risks in data flow are analyzed, and a comprehensive risk scoring model is constructed through deep learning, specifically: Based on the dynamic data flow model, potential risks in data flow are analyzed, classification extraction is performed through SVM, and a comprehensive risk score model is constructed; The comprehensive risk score model is specifically: ； wherein R represents the comprehensive risk score; f i (z i ) represents the fitting mapping function of the i-th risk factor; z i represents the standardized data; β represents the adjustment factor, β ∈ [0, 1].

6. The data flow security supervision method based on the big data platform according to claim 5, characterized in that: Specifically, A first risk threshold Q1 is set; A second risk threshold Q2 is set; An early warning device is provided; When the comprehensive risk score R is greater than or equal to the first risk threshold Q1, it is classified as high risk; When the comprehensive risk score R is less than the first risk threshold Q1 but greater than or equal to the second risk threshold Q2, it is classified as medium risk; When the comprehensive risk score R is less than the second risk threshold Q2, it is classified as low risk; According to the classified risk level, the pre-prepared management strategy is used for early warning and management.

7. A data flow security supervision system based on a big data platform, applying a data flow security supervision method based on a big data platform according to any one of claims 1-6, characterized in that: The data flow safety supervision system comprises a data collection module, a feature selection module, a dynamic data flow module, a risk assessment module and an early warning and management module; The data collection module is used to collect transmission process data and user operation data, the transmission process data includes network traffic data and encryption information data; the user operation data includes user behavior data, operation frequency and permission information data; The feature selection module is used to analyze the collected data, extract and standardize risk factor data; The dynamic data flow module is used to construct a data full life cycle graph and dynamically analyze user behavior flow; The risk assessment module is used to construct a comprehensive risk score model based on deep learning for real-time risk assessment; The early warning and management module is used to set a risk threshold, divide a risk level according to the risk threshold, and implement corresponding early warning and management strategies for different risk levels. 8.The data flow security supervision system based on a big data platform of claim 7, wherein: The data collection module comprises a software adaptation unit and a data collection unit; The software adaptation unit is used to install and test data collection software; The data collection unit is used to collect transmission process data and user operation data.

9. The data flow safety supervision system based on the big data platform according to claim 8, characterized in that: The early warning and management module provides a user interaction interface, supports visual display of real-time comprehensive risk score, displays historical management strategies, and provides a reference for management personnel.

Images