Data leakage protection method and device, equipment and storage medium
By obtaining and processing data flows in real time and processing them in parallel, using network application firewalls and distributed message processing clusters for data leakage risk detection, the problem of insufficient timeliness and automated responses in traditional solutions is solved, and efficient protection of data leakage is achieved.
Patent Information
- Application Number
- CN202411001145.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-24
AI Technical Summary
Traditional data leakage protection solutions have shortcomings in timeliness and automated responses, and cannot effectively prevent data leakage.
By obtaining the data stream to be detected in real time, it is sent to the network application firewall and distributed message processing cluster in parallel, using both to perform real-time data leakage risk detection, and performing data blocking operations when risks are detected.
It improves the real-time and accuracy of data breach risk detection, ensures that data is not leaked or abused, and achieves efficient processing and timely response to large-scale data.
Smart Images

Figure CN118965339A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data security technology, and in particular to a data leakage protection method, device, equipment and storage medium. Background Art
[0002] In the era of digital transformation, data leakage has become a serious security threat, which may lead to serious consequences such as leakage of user privacy and commercial secrets.
[0003] Traditional security solutions usually use technologies such as intrusion detection systems (IDS), intrusion prevention systems (IPS), and application programming interface (API) risk monitoring platforms to deal with data leakage issues.
[0004] However, the main problem with traditional solutions when it comes to data leakage protection is that the timeliness of data analysis cannot be guaranteed, and after risks are discovered, they can only rely on human intervention, which fails to effectively protect data. Summary of the invention
[0005] The embodiments of the present application provide a data leakage protection method, apparatus, device and storage medium, which can solve the technical problem of how to prevent data leakage efficiently and timely.
[0006] In a first aspect, an embodiment of the present application provides a data leakage protection method, including:
[0007] Acquire the data stream to be detected in real time.
[0008] The data stream to be detected is sent in parallel to the first detection system and the distributed message processing cluster; wherein the distributed message processing cluster and the second detection system are configured with a subscription relationship, and the second detection system is used to extract the data stream to be detected received by the distributed message processing cluster based on the subscription relationship.
[0009] The first detection system and / or the second detection system are used to perform real-time data leakage risk detection on the data stream to be detected in parallel.
[0010] When either the first detection system or the second detection system detects that there is a risk of data leakage in the data flow to be detected, a data blocking operation is performed.
[0011] In one embodiment, the first detection system is a network application firewall, and the second detection system is a data risk identification system. The first detection system and / or the second detection system are used to perform real-time data leakage risk detection on the data flow to be detected in parallel, including: using the network application firewall to perform real-time data leakage risk detection on the data flow to be detected to obtain a first detection result; using the data risk identification system to perform real-time data leakage risk detection on the data flow to be detected according to preset risk identification rules to obtain a second detection result, and the preset risk identification rules include at least one of the following: setting regular sensitive data identification rules, access request frequency analysis rules, and access volume and access time analysis rules.
[0012] In one embodiment, the first detection system includes a first risk identification device, and the second detection system includes a second risk identification device. The data risk identification system is used to perform real-time data leakage risk detection on the data flow to be detected according to preset risk identification rules to obtain a second detection result, including: using a consistent hashing algorithm to determine the virtual node number of the second risk identification device; performing a hash calculation on the number name of the first risk identification device and the address name of the user end to obtain an association table, the association table is used to record the association relationship between the first risk identification device, the second risk identification device and the user end, and the user end is used to indicate the source of the data flow to be detected; according to the association table, the data flow to be detected is distributed to the corresponding virtual node of the second risk identification device, and real-time data leakage risk detection is performed to obtain a second detection result.
[0013] In one embodiment, a data risk identification system is used to perform real-time data leakage risk detection on a data stream to be detected according to preset risk identification rules to obtain a second detection result, including: retrieving historical detection data streams from the data risk identification system; and performing parallel analysis on the historical detection data streams and the data stream to be detected according to preset risk identification rules to obtain a second detection result.
[0014] In one embodiment, real-time acquisition of the data stream to be detected includes: configuring preset data acquisition rules, the preset data acquisition rules including at least one of the following: target name, target port, target protocol; according to the preset data acquisition rules, using the network application firewall to obtain the data stream to be detected in real time.
[0015] In one embodiment, when it is detected that the data flow to be detected has a data leakage risk, a data blocking operation is performed, including: when the first detection result has a data leakage risk, using a network application firewall to terminate the network connection of the user terminal address with the data leakage risk.
[0016] In one embodiment, when there is a risk of data leakage in the second detection result, based on the subscription relationship configured between the second detection system and the distributed message processing cluster, the second detection system sends an early warning notification to the distributed message processing cluster, and the distributed message processing cluster is used to instruct the network application firewall to terminate the network connection of the user terminal address with the risk of data leakage according to the early warning notification.
[0017] In a second aspect, an embodiment of the present application provides a data leakage protection device, which has the function of implementing the method in the first aspect or any possible implementation thereof. Specifically, the device includes a unit that implements the method in the first aspect or any possible implementation thereof.
[0018] In one embodiment, the device comprises:
[0019] The acquisition unit is used to acquire the data stream to be detected in real time.
[0020] A processing unit is used to send the data stream to be detected to the first detection system and the distributed message processing cluster in parallel; wherein the distributed message processing cluster and the second detection system are configured with a subscription relationship, and the second detection system is used to extract the data stream to be detected received by the distributed message processing cluster based on the subscription relationship.
[0021] The processing unit is further configured to use the first detection system and / or the second detection system to perform real-time data leakage risk detection on the data stream to be detected in parallel.
[0022] The processing unit is further configured to perform a data blocking operation when either the first detection system or the second detection system detects that there is a risk of data leakage in the data stream to be detected.
[0023] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements a method according to any one of the implementation modes of the first aspect.
[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer device, the computer device implements the method of any one of the implementation methods of the first aspect above.
[0025] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device executes a method of any one of the implementation modes of the first aspect above.
[0026] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by acquiring the data stream to be detected in real time, the latest data stream can be obtained in time, ensuring the real-time and accuracy of data leakage risk detection; by sending the data stream to be detected to the first detection system and the distributed message processing cluster in parallel, the processing efficiency of the system can be improved; the second detection system extracts the data stream of the distributed message processing cluster based on the subscription relationship, which can ensure the integrity and consistency of the data; the first detection system and / or the second detection system can improve the detection efficiency and timeliness by performing data leakage risk detection in parallel, and can process relatively large amounts of data to be detected at the same time; when a data leakage risk is detected, a data blocking operation is performed, which can respond to the data leakage risk in time and ensure that the data is not leaked or abused. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a schematic diagram of a data leakage scenario provided in an embodiment of the present application.
[0028] Figure 2 It is a flow chart of a data leakage protection method provided in an embodiment of the present application.
[0029] Figure 3 It is a schematic diagram of a hash ring provided in an embodiment of the present application.
[0030] Figure 4 It is an interactive flow chart of a data leakage protection method provided in an embodiment of the present application.
[0031] Figure 5 It is a structural schematic diagram of a data leakage protection device provided in an embodiment of the present application.
[0032] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] Data leakage refers to the unauthorized or unintentional disclosure of sensitive data to unauthorized third parties. With the digital transformation of enterprises, the increase in various business access channels, such as web pages, mobile applications and APIs, provides more opportunities for data leakage.
[0034] Combine the following Figure 1 Let’s explain a data leakage scenario in detail.
[0035] Figure 1 It is a schematic diagram of a data leakage scenario provided in an embodiment of the present application.
[0036] like Figure 1As shown, when a user accesses the network, if there are security vulnerabilities in the application the user is using, attackers can exploit these vulnerabilities to bypass security measures and obtain the user's private data.
[0037] In response to the above problems, this application proposes a data leakage protection method that can efficiently and real-time discover data leakage risks and block data in a timely manner.
[0038] In order to further illustrate the technical solution of the present application, specific embodiments are provided below.
[0039] Figure 2 It is a flow chart of a data leakage protection method provided in an embodiment of the present application.
[0040] like Figure 2 As shown, the above method includes the following steps S201 to S204.
[0041] S201. Acquire the data stream to be detected in real time.
[0042] A data stream refers to a series of continuous data packets or information sequences that are transmitted between networks or systems in the form of a stream. It can be understood that the data stream to be detected refers to the data stream that needs to be monitored and detected to ensure that it does not contain sensitive information or data leakage risks.
[0043] S202: Send the data stream to be detected to the first detection system and the distributed message processing cluster in parallel.
[0044] The distributed message processing cluster and the second detection system are configured with a subscription relationship, and the second detection system is used to extract the data stream to be detected received by the distributed message processing cluster based on the subscription relationship.
[0045] The first detection system and the second detection system refer to two independent data detection systems or software, which are used to perform security detection on the data stream to be detected.
[0046] As an example but not limitation, the first detection system may refer to a Web Application Firewall (WAF); the second detection system may be a processing system that performs risk analysis on network data flows in a time sequence. The system may identify unauthorized access, sensitive data identification, excessive sensitive data returned at one time, sensitive data page turning requests, etc., that is, it may identify the risks of a single time sequence flow, and may also determine whether there is a risk of data leakage in the entire protection system based on the situation.
[0047] Web Application Firewall (WAF) identifies and protects malicious features of business traffic from websites, apps, APIs, etc. After cleaning and filtering the traffic, it returns normal and safe traffic to the server to prevent malicious intrusions on the website server that may cause performance anomalies, thereby ensuring the business security and data security of the website. WAF focuses on attack protection for network requests, and does not have the ability to identify data types, data flows, and data leakage risks. For example, it cannot detect: unauthorized data acquisition, excessive data acquisition at one time, abnormal flow of sensitive data, remote login to acquire large amounts of data, and abnormal data outflow from abroad.
[0048] It is understandable that the use of WAF for data collection and blocking can ensure the timeliness of data extraction because WAF performs security checks in a time-series manner when processing network data packets, that is, it will release the request packet only after it is determined that there is no risk; WAF is accessed through a transparent proxy or reverse proxy, and has absolute control over network blocking. When data risks are discovered, it can be blocked in real time and subsequent access requests can be restricted. When protecting application security at the border, WAF will be deployed to protect the network security of the application layer to prevent hackers from breaking into the network through the application layer.
[0049] As an example and not a limitation, a distributed message processing cluster may refer to a Kafka message processing system. Kafka is a distributed streaming message processing system that can extract and process streaming data in real time. Streaming data refers to data that can be continuously generated by thousands of data sources, and data records can usually be sent at the same time. The streaming platform needs to process this continuously flowing data and process it step by step in sequence. Kafka's time-based, high-performance processing capabilities are used to transfer and store data. On the one hand, it can ensure that the operation of the original WAF system is not affected; on the other hand, it can allow the risk identification cluster to perform efficient subscription processing to ensure the real-time and efficiency of data processing.
[0050] It can be understood that the subscription relationship means that the second detection system registers a subscription with the distributed message processing cluster so as to receive the data stream received by the cluster. Once the subscription relationship is established, the second detection system can obtain the data stream in the distributed message processing cluster in real time and perform relevant processing.
[0051] Through the subscription relationship, the second detection system can receive the data stream in the distributed message processing cluster in real time to ensure data synchronization and consistency.
[0052] S203: Use the first detection system and / or the second detection system to perform real-time data leakage risk detection on the data stream to be detected in parallel.
[0053] Data leakage risk detection can be to detect whether sensitive data such as personal identity information, financial information, intellectual property information, etc. is leaked in the data stream. As an example but not limitation, the data stream to be detected can be analyzed and detected by data classification, keyword matching, regular expressions, etc. to determine whether there is data leakage.
[0054] It can be understood that performing real-time data leakage risk detection on the data stream to be detected in parallel by the first detection system and / or the second detection system means that the two detection systems can independently detect the data stream. If both systems indicate that there may be a leakage risk, the data leakage can be more confirmed. At the same time, parallel processing can save time, shorten the time for detecting the data stream to be processed, and help to discover potential data leakage risks more quickly.
[0055] S204: When either the first detection system or the second detection system detects that there is a risk of data leakage in the data stream to be detected, a data blocking operation is performed.
[0056] Data blocking means taking immediate measures to interrupt data transmission or access when data leakage risks are detected to prevent sensitive information from being leaked. For example, when an attacker is detected trying to download internal database information to the outside, the system can automatically block this behavior to protect the security of user data.
[0057] The above-mentioned data leakage protection method can obtain the latest data stream in time by acquiring the data stream to be detected in real time, thereby ensuring the real-time and accuracy of data leakage risk detection; by sending the data stream to be detected to the first detection system and the distributed message processing cluster in parallel, the processing efficiency of the system can be improved, and the data can be backed up to the distributed message processing cluster at the same time to ensure the reliability and persistence of the data; the second detection system extracts the data stream of the distributed message processing cluster based on the subscription relationship, thereby ensuring the integrity and consistency of the data, while reducing invasive operations on the original data; the first detection system and / or the second detection system can improve the detection efficiency and timeliness by performing data leakage risk detection in parallel, thereby ensuring that potential data leakage risks can be discovered as early as possible; when a data leakage risk is detected, a data blocking operation is performed, thereby being able to respond to the data leakage risk in a timely manner, taking effective measures to protect data security, and ensuring that the data is not leaked or abused.
[0058] In one embodiment, the first detection system is a network application firewall, and the second detection system is a data risk identification system. The first detection system and / or the second detection system are used to perform real-time data leakage risk detection on the data flow to be detected in parallel, including: using the network application firewall to perform real-time data leakage risk detection on the data flow to be detected to obtain a first detection result; using the data risk identification system to perform real-time data leakage risk detection on the data flow to be detected according to preset risk identification rules to obtain a second detection result, and the preset risk identification rules include at least one of the following: setting regular sensitive data identification rules, access request frequency analysis rules, and access volume and access time analysis rules.
[0059] In one implementation, the web application firewall uses the ModSecurity module or other risk identification technologies to identify attacks on the data stream to be detected, such as anomaly detection based on machine learning, for identifying and responding to various network security threats and risks.
[0060] ModSecurity is an open source web application firewall that can be used to detect and block malicious attacks in web applications, such as SQL injection, cross-site scripting attacks, etc. ModSecurity uses a rule engine to analyze HTTP requests and responses, identify potential malicious behaviors, and take corresponding defensive measures such as interception, logging, and alarming according to predefined rule sets.
[0061] For example, if a website is being attacked by malicious SQL injection, ModSecurity can configure corresponding rules to detect and block such attacks. When a request containing malicious SQL statements is sent to the website, ModSecurity can analyze the request content, identify the SQL injection attack characteristics, and intercept or log the request according to the pre-set rules, thereby protecting the website from attacks.
[0062] The following is a detailed introduction to several preset risk identification rules used by the second detection system.
[0063] (1) Setting regular sensitive data identification rules
[0064] In the regular sensitive data identification rules, some common sensitive data patterns can be set, such as ID card numbers, credit card numbers, mobile phone numbers, etc. Regular expressions are used to match the content in the data stream. Once a matching sensitive data pattern is found, a data leakage risk alert will be triggered. For example, a regular expression can be set to match all 15-digit and 18-digit ID card numbers. If the data stream contains content that matches the regular expression, the second detection system will find potential data leakage risks.
[0065] (2) Access request frequency analysis rules
[0066] Access request frequency analysis rules can be used to detect whether a user or system has abnormal access frequency to data. For example, a rule can be set that if a user frequently accesses a protection site within a short period of time, exceeding the normal access frequency range, it will be considered a data leakage risk. This rule can help detect possible malicious access behavior or data leakage incidents.
[0067] (3) Rules for analyzing visit volume and visit time
[0068] The access volume and access time analysis rules can be used to detect the regularity and abnormality of data access. For example, you can set a rule to monitor the total amount of data accessed every day. If the number of accesses on a certain day is significantly higher than the average, or there are a large number of accesses during non-working hours, there may be a risk of data leakage. By analyzing the regularity of access volume and access time, you can detect abnormal situations in a timely manner and take corresponding measures.
[0069] In combination with the above, the first detection system identifies data leaks by utilizing the Web application firewall risk identification module, and the second detection system identifies data leaks by utilizing preset risk identification rules such as sensitive data identification rules in a regular manner, access request frequency analysis rules, access volume and access time analysis rules. The combined use of these two methods can improve the accuracy and efficiency of data leakage detection, timely discover and respond to potential data leakage risks, and protect the security and integrity of data.
[0070] In one embodiment, the first detection system includes a first risk identification device, and the second detection system includes a second risk identification device. The data risk identification system is used to perform real-time data leakage risk detection on the data flow to be detected according to preset risk identification rules to obtain a second detection result, including: using a consistent hashing algorithm to determine the virtual node number of the second risk identification device; performing a hash calculation on the number name of the first risk identification device and the address name of the user end to obtain an association table, the association table is used to record the association relationship between the first risk identification device, the second risk identification device and the user end, and the user end is used to indicate the source of the data flow to be detected; according to the association table, the data flow to be detected is distributed to the corresponding virtual node of the second risk identification device, and real-time data leakage risk detection is performed to obtain a second detection result.
[0071] Combine the following Figure 3 Let me introduce it in detail.
[0072] Figure 3 It is a schematic diagram of a hash ring provided in an embodiment of the present application.
[0073] like Figure 3As shown, it is assumed that there are 3 WAF devices (first risk identification devices), and the ID codes of the WAF devices are IDF1, IDF2, and IDF3 respectively; there are 3 data processing servers (second risk identification devices), and the codes of the data processing servers are A, B, and C respectively. Among them, the performance of the data processing servers can be consistent, or the performance of the data processing servers can also be set according to the weight, which is not limited here.
[0074] In the initial distribution, the consistent hashing algorithm is used to distribute the nodes in 2 32 On the ring of the same data center, assuming that the number of virtual nodes of each data processing server is increased to 3, we can get:
[0075] Node A is numbered as a virtual node, and the virtual node numbers are: A-01, A-02, A-03;
[0076] Node B is numbered as a virtual node, and the virtual node numbers are: B-01, B-02, B-03;
[0077] Node C is numbered as a virtual node, and the virtual node numbers are: C-01, C-02, C-03.
[0078] Next, record the value of the virtual node of each data processing server, and then perform hash calculation based on the WAF ID and the last segment of the user's IP address to record the association table, as shown in Table 1 below.
[0079] Table 1
[0080] Data risk identification server WAF+IP encoding group A 1、4、7、11...... B 2、5、8、12...... C 3、6、9、13......
[0081] As can be seen from Table 1, according to the number of data processing servers, the data streams to be detected are evenly distributed, and each data processing server only needs to subscribe according to the distributed value. In other words, each data processing server will subscribe to the data stream corresponding to it. For example, server A will subscribe to the data stream assigned to node A after hash calculation, server B will subscribe to the data stream assigned to node B after hash calculation, and so on.
[0082] The following describes the mechanism when the WAF device or data processing server fails or changes.
[0083] Assuming that a new WAF device is added, you only need to perform hash calculation based on the WAF device's code + IP tail number, and associate it with the data risk identification node, and the pressure will still be balanced.
[0084] Assuming that a WAF device stops working, the original allocation does not need to be modified. The subscription just means that there is no data to be processed, which does not affect the processing of the entire service cluster.
[0085] Assuming that a new data processing server is added, the association table is recalculated. When the original user IP is disconnected or not connected, the new server is subscribed, so that the data of the original processing server can be smoothly transitioned).
[0086] Assuming that the data processing server is down, the association table is recalculated and the newly added service processing subject is subscribed immediately.
[0087] It can be understood that by performing parallel data processing on multiple devices and ensuring a high degree of concurrency in data processing, system resources can be fully utilized to maximize data processing efficiency; secondly, by using the data processing server number, WAFID and user-side IP to establish an association table, and distributing data based on the association table, it is ensured that all data from the same user-side IP is managed by the same data risk identification system, maintaining the consistency and accuracy of the processing results; finally, the design incorporates a fault switching (change) mechanism, which can automatically redistribute tasks when individual data risk identification systems encounter abnormalities, thereby ensuring the continuity of data processing and the high availability of the system, and achieving flexible rebalancing of processing loads.
[0088] In this way, balanced processing of the data streams to be detected is ensured. Each data processing server only needs to focus on the data streams related to itself and will not be disturbed by the data processed by other servers. The performance of each data processing server can be effectively utilized to achieve distributed and efficient data risk identification and processing.
[0089] In one embodiment, a data risk identification system is used to perform real-time data leakage risk detection on a data stream to be detected according to preset risk identification rules to obtain a second detection result, including: retrieving historical detection data streams from the data risk identification system; and performing parallel analysis on the historical detection data streams and the data stream to be detected according to preset risk identification rules to obtain a second detection result.
[0090] As an example but not limitation, historical detection data streams may be extracted from a database, log file, or other data storage medium, and the data may contain information such as known malicious patterns, attack features, and the like.
[0091] Then, according to the preset risk identification rules and the relevant contents of the association table mentioned above, the data stream to be detected and the historical detection data stream are simultaneously distributed to the corresponding second risk identification device to perform real-time data leakage risk detection to obtain a second detection result.
[0092] It is understandable that the newly received data and historically processed data are analyzed in parallel, and information from different time periods is integrated to comprehensively assess the potential risk of data leakage. Through this data fusion analysis over a time span, the accuracy and comprehensiveness of the judgment are ensured.
[0093] In one embodiment, real-time acquisition of the data stream to be detected includes: configuring preset data acquisition rules, the preset data acquisition rules including at least one of the following: target name, target port, target protocol; according to the preset data acquisition rules, using the network application firewall to obtain the data stream to be detected in real time.
[0094] As an example and not a limitation, data acquisition rules can be configured in WAF to ensure that WAF can capture and process the required data flow. For example, define the target name (user end or server end), select the target protocol (such as HTTP, HTTPS), set the target port, etc. It is understood that the specific data acquisition rules can be customized according to the actual situation.
[0095] Then, WAF captures the transmitted data flow through the network interface or proxy according to the configured data acquisition rules.
[0096] Through the above steps, WAF can capture and organize the data flow to be detected, thereby providing support for subsequent data analysis, security detection and protection.
[0097] In one embodiment, when it is detected that the data flow to be detected has a data leakage risk, a data blocking operation is performed, including: when the first detection result has a data leakage risk, using a network application firewall to terminate the network connection of the user terminal address with the data leakage risk.
[0098] It can be understood that once WAF determines that there is a risk of data leakage, it will accurately locate the associated network link based on the IP address of the corresponding user terminal and immediately terminate the network connection.
[0099] In one example, you can also implement temporary access restrictions or permanently blacklist the IP address based on preset policies, effectively preventing potential threats in the future and consolidating the network security line of defense.
[0100] In one embodiment, when there is a risk of data leakage in the second detection result, based on the subscription relationship configured between the second detection system and the distributed message processing cluster, the second detection system sends an early warning notification to the distributed message processing cluster, and the distributed message processing cluster is used to instruct the network application firewall to terminate the network connection of the user terminal address with the risk of data leakage according to the early warning notification.
[0101] When it is determined that there is a risk of data leakage, the second detection system is responsible for constructing a blocking instruction and pushing it to the Kafka message queue, thereby initiating a network disconnection operation. Once Kafka receives the blocking instruction, it immediately notifies the subscribed related WAF to pull data. Subsequently, the WAF quickly obtains the data and executes emergency blocking measures for the network connection to ensure the immediacy and effectiveness of the response.
[0102] Combine the following Figure 4 Let’s give an overall introduction to the data leakage protection methods mentioned above.
[0103] Figure 4 It is an interactive flow chart of a data leakage protection method provided in an embodiment of the present application.
[0104] like Figure 4 As shown, the method is divided into two stages, namely the preparation stage and the processing stage.
[0105] The preparation phase is mainly to prepare for high-performance operation in advance, which mainly includes the following contents:
[0106] S401. WAF uses connection pool technology to establish and maintain long-term connections with the Kafka cluster.
[0107] The connection pool is used to manage the reuse and management of database connections, network connections, or other resource connections. Here, the connection pool is used to manage the long-term connection between the WAF (Web Application Firewall) and the Kafka cluster to improve the reusability and efficiency of the connection.
[0108] Specifically, when the system starts, a certain number of connections are created and stored in the connection pool. These connections can be connections established with the Kafka cluster for data transmission and communication. When the system needs to communicate with the Kafka cluster, it obtains an available connection from the connection pool. If there is no available connection in the connection pool, you can choose to wait or create a new connection (depending on the specific configuration). The system uses the connection to communicate with the Kafka cluster, transmit data and receive responses. After the communication is completed, the connection is returned to the connection pool for reuse next time.
[0109] By using connection pool technology, the overhead of establishing and closing connections for each communication can be reduced, improving system performance and resource utilization. The connection pool can also limit the number of connections to avoid the problem of insufficient system resources due to too many connections, and dynamically expand connections when needed to ensure the optimal balance of resource usage while maintaining high performance.
[0110] S402, WAF subscription blocking message.
[0111] WAF uses a subscription model to interact with Kafka to ensure that it can obtain its own blocking instructions immediately, thereby quickly interrupting the connection of potential risk users and effectively curbing the risk of data leakage. Specifically, the data risk processing system is responsible for pushing messages to Kafka, and these messages are designed based on Topics (specific topics: killLink+WAFID) to achieve accurate target communication and instruction execution, enhancing the timeliness and accuracy of security response.
[0112] S403. Push risk rules.
[0113] The rule setting system predefines the rules for risk identification so that the data risk identification system can accurately judge various risks, such as setting regular methods for sensitive data identification, analyzing access requests by frequency, analyzing data traffic by access volume and time, etc.
[0114] It can be understood that an independent rule management module is designed, which supports dynamic rule editing function. It can flexibly adjust the verification rules according to actual needs without interfering with the normal operation of the online system, realize accurate local deployment and verification, and greatly improve the flexibility and stability of the system.
[0115] The risk identification system pre-loads a customized set of rules and pre-builds an optimized processing flow module matrix to ensure that efficient verification operations can be immediately carried out in each parallel module when data flows in, quickly identifying whether there is a potential threat of data leakage, greatly improving the timeliness and accuracy of risk detection.
[0116] It can be understood that the distributed architecture design of the data risk identification system not only ensures a high level of scalability and excellent performance, but also provides a strong elastic foundation for the system. This enables seamless integration of container orchestration technology when deployed in a cloud environment, enabling dynamic expansion and contraction of resources to cope with changing business needs and maximize the flexibility and efficiency of cloud resources.
[0117] S404: Subscribe to data.
[0118] The risk identification system uses the connection pool mechanism to establish a stable connection with Kafka, and pre-subscribes (and subsequently dynamically adjusts) the data streams that need to be processed for each module.
[0119] The subscription logic is achieved by integrating the WAF ID, the client identifier (IP) and the risk identification system performance indicator weights (CPU, memory, network resources) through a ring distribution algorithm (combined with the above Figure 3 to achieve smart subscription.
[0120] It can be understood that by distributing data through the WAF ID and user-side identifier (IP), it can be ensured that data from the same source is consistently processed by the same detection system, thereby enhancing the accuracy of risk identification. Secondly, by using WAF, Kafka, and the data risk identification system to identify data leakage risks in parallel, it is possible to optimize resource scheduling, fully tap the processing potential of the cluster, and ensure that data is processed in real time with ultra-low latency. Finally, by using the association table method mentioned above to distribute data, even if a node in the cluster suddenly fails, it can ensure that service continuity is not affected, thereby improving the overall stability and resilience of the system.
[0121] The processing stage mainly involves efficient verification of all application layer network data (user requests, server responses), and immediate blocking when problems are found. It mainly includes the following contents:
[0122] S405: Receive a network data packet.
[0123] In this step, WAF collects data from the network (user side or server side) and organizes the data flow to ensure that it can be identified and returned to the corresponding terminal in the subsequent processing flow.
[0124] S406A, data risk management.
[0125] WAF uses ModSecurity or other risk identification technologies to detect data leakage risks.
[0126] S406B, synchronous data push.
[0127] The data is pushed to Kafka. The topic (Topic) of the push message contains key elements such as the data feature identifier (DATA field), WAF unique identifier (WAFID) and user source IP address, ensuring the targeted information transmission and accurate matching of the processing flow.
[0128] That is, after the WAF receives and marks the data, it simultaneously executes steps S406A and S406B on the data for parallel processing.
[0129] Since the data is not disassembled here, smart pointer technology is used to distribute the data in parallel without copying the data, avoiding the consumption of memory, CPU and time, and further improving the real-time processing.
[0130] S407: Obtain subscription data.
[0131] When Kafka receives data, it immediately triggers the notification mechanism according to the previously set subscription configuration, and notifies the associated data risk identification system in a targeted manner. Subsequently, the system responds quickly and extracts data, ensuring seamless connection and efficient start-up of the risk assessment process.
[0132] S408. Data risk processing.
[0133] In this step, the system analyzes the newly received data and the historically processed data in parallel according to the preset risk rule framework, comprehensively considering both aspects of information to comprehensively assess the potential data leakage risk. Through this data fusion analysis over a time span, the accuracy and comprehensiveness of the judgment are ensured. Once a risk sign is identified, the system immediately triggers the preset response mechanism, which may include issuing an alarm or activating step S409, disconnecting the network connection and synchronizing the alarm operation, and quickly and effectively curbing security threats.
[0134] S409: There is a risk. Block the network.
[0135] When step S408 determines that network blocking is required, the system will activate the process, which is responsible for constructing the blocking instruction and efficiently pushing it to the Kafka message queue, thereby starting the network disconnection operation.
[0136] S410, there is a risk, block the network.
[0137] Once Kafka receives a blocking instruction, it immediately notifies the relevant subscribed WAF to pull data. Subsequently, the WAF quickly obtains the data and executes emergency blocking measures for the network connection to ensure the immediacy and effectiveness of the response.
[0138] S411, block or return.
[0139] Once WAF receives risk warning information, it will accurately locate the associated network link based on the user IP address contained in the message and immediately terminate the session. At the same time, according to the preset strategy, it can implement temporary access restrictions or permanently add the IP to the blacklist, effectively preventing potential threats in the future, consolidating the network security line of defense, and ensuring immediate response to any identified security threats (including attacks or data leaks).
[0140] The above mainly introduces a data leakage protection method of an embodiment of the present application in conjunction with the accompanying drawings. It should be understood that although the steps in the flowcharts involved in the embodiments described above are displayed in sequence, these steps are not necessarily performed in sequence in the order shown in the figure. Unless otherwise specified in this article, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps. The following is an introduction to a device of an embodiment of the present application in conjunction with the accompanying drawings. For the sake of brevity, when introducing the device below, it will be appropriately omitted, and the relevant content can refer to the relevant description in the method above, and will not be repeated.
[0141] Figure 5 It is a structural schematic diagram of a data leakage protection device provided in an embodiment of the present application.
[0142] like Figure 5 As shown, the device 1000 includes the following units.
[0143] The acquisition unit 1001 is used to acquire the data stream to be detected in real time.
[0144] Processing unit 1002 is used to send the data stream to be detected to the first detection system and the distributed message processing cluster in parallel; wherein the distributed message processing cluster and the second detection system are configured with a subscription relationship, and the second detection system is used to extract the data stream to be detected received by the distributed message processing cluster based on the subscription relationship.
[0145] The processing unit 1002 is further configured to use the first detection system and / or the second detection system to perform real-time data leakage risk detection on the data stream to be detected in parallel.
[0146] The processing unit 1002 is further configured to perform a data blocking operation when any one of the first detection system and the second detection system detects that there is a risk of data leakage in the data stream to be detected.
[0147] In one implementation, the device 1000 further includes a storage unit 1003, which can be used to store instructions and / or data, so as to implement the method in the above embodiment.
[0148] In one embodiment, the storage unit 1003 is also used to perform real-time data leakage risk detection on the data stream to be detected in parallel using the first detection system and / or the second detection system, including: the first detection system uses the Web application firewall risk identification module to perform real-time data leakage risk detection on the data stream to be detected to obtain a first detection result; the second detection system performs real-time data leakage risk detection on the data stream to be detected according to preset risk identification rules to obtain a second detection result, and the preset risk identification rules include at least one of the following: sensitive data identification rules in a regular manner, access request frequency analysis rules, and access volume and access time analysis rules.
[0149] It is understandable that the first detection system and the second detection system can asynchronously perform data leakage risk detection, improve the efficiency and performance of the system, and effectively prevent the risks caused by loopholes and imperfect rules. Through different detection rules and methods, the coverage of various data leakage risks can be increased to avoid potential security loopholes.
[0150] In one embodiment, the storage unit 1003 is also used to determine the virtual node number of the second risk identification device using a consistent hashing algorithm; perform hash calculation on the number name of the first risk identification device and the address name of the user terminal to obtain an association table, the association table is used to record the association relationship between the first risk identification device, the second risk identification device and the user terminal, and the user terminal is used to indicate the source of the data stream to be detected; distribute the data stream to be detected to the corresponding second risk identification device according to the association table, perform real-time data leakage risk detection, and obtain a second detection result.
[0151] It can be understood that the consistent hashing algorithm can dynamically distribute data streams to different risk identification devices according to data characteristics, achieve load balancing, avoid overloading a certain device, and improve the overall performance and efficiency of the system. Distributing data streams to corresponding risk identification devices according to the association table can ensure that the data stream is sent to the device that is most suitable for detection, thereby improving detection efficiency and accuracy. By recording the relationship between devices in the association table, the mapping relationship between devices can be adjusted at any time to achieve dynamic configuration and management between devices, thereby improving the flexibility and scalability of the system. By recording the relationship between devices in the association table, even if a certain device fails or fails, the system can quickly distribute the data stream to other backup devices for processing, thereby ensuring the smooth progress of the data detection task. By establishing an association table, it can be ensured that the data processing process on different devices is consistent, thereby avoiding inconsistent results caused by differences in data processing on different devices.
[0152] In one embodiment, the storage unit 1003 is also used to use the second detection system to perform real-time data leakage risk detection on the data stream to be detected according to preset risk identification rules to obtain a second detection result, including: retrieving historical detection data streams from the second detection system; and performing parallel analysis on the historical detection data streams and the data stream to be detected according to preset risk identification rules to obtain a second detection result.
[0153] It can be understood that by analyzing historical detection data streams and data streams to be detected in parallel, risks in data streams can be detected more carefully, improving the accuracy and efficiency of detection. At the same time, historical data comparison can be used to better understand the current data stream situation and provide more comprehensive risk identification services.
[0154] In one embodiment, the acquisition unit 1001 is also used to acquire the data stream to be detected in real time, including: configuring preset data acquisition rules, the preset data acquisition rules include at least one of the following: target name, target port, target protocol; according to the preset data acquisition rules, using the network application firewall to acquire the data stream to be detected in real time.
[0155] It can be understood that by configuring data acquisition rules, the data streams that need to be detected can be accurately acquired. This ensures that the system acquires key data for risk identification and improves system operation efficiency and accuracy.
[0156] In one embodiment, the processing unit 1002 is also used to perform a data blocking operation when it is detected that the data flow to be detected has a data leakage risk, including: when the first detection result has a data leakage risk, using the network application firewall to terminate the network connection of the user terminal address with the data leakage risk.
[0157] It can be understood that when the first detection result shows that there is a risk of data leakage, timely terminating the network connection of the risky user address can effectively prevent the continued occurrence of data leakage and protect data security.
[0158] In one embodiment, the processing unit 1002 is also used for, when the second detection result presents a risk of data leakage, the second detection system sends a warning notification to the distributed message processing cluster; the distributed message processing cluster instructs the network application firewall to terminate the network connection of the user terminal address presenting a risk of data leakage according to the warning notification.
[0159] It can be understood that when the second detection result shows that there is a risk of data leakage, the system can send a warning notification to the distributed message processing cluster in a timely manner and instruct the first detection system to terminate the network connection of the client address at risk. This can achieve a rapid response and processing of data leakage incidents, avoiding further data leakage and causing greater losses. Through timely warning and automated processing, the efficiency and accuracy of responding to data leakage incidents can be improved.
[0160] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0161] Figure 6 Schematic diagram of the structure of the computer device provided in the embodiment of the present application. Figure 6 As shown, the computer device 6000 of this embodiment includes: at least one processor 6100 ( Figure 6 Only one) processor, memory 6200, and a computer program 6210 stored in the memory 6200 and executable on at least one processor 6100 are shown. When the processor 6100 executes the computer program 6210, the computer device implements the steps in the above-mentioned embodiments.
[0162] The processor 6100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0163] The memory 6200 may be an internal storage unit of the computer device 6000 in some embodiments, such as a hard disk or memory of the computer device 6000. The memory 6200 may also be an external storage device of the computer device 6000 in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 6000. Further, the memory 6200 may also include both an internal storage unit of the computer device 6000 and an external storage device. The memory 6200 is used to store an operating system, an application program, a boot loader (Boot Loader) data, and other programs, such as a program code of a computer program, etc. The memory 6200 may also be used to temporarily store data that has been output or is to be output.
[0164] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0165] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer device, the computer device can implement the steps in the above-mentioned method embodiments.
[0166] The embodiment of the present application provides a computer program product. When the computer program product is executed on a computer device, the computer device can implement the above-mentioned methods.
[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the computer device can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0168] It should be understood that the size of the sequence number of each step in the above-described embodiment does not mean the order of execution, and the order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application. In the description, for the purpose of explanation rather than for limitation, specific details such as specific system structure, technology and the like are proposed, so as to thoroughly understand the embodiment of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.
[0169] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0170] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0171] Additionally, in the specification of the present application and the description of the appended claims, the terms “include,” “comprise,” “have,” and their variations all mean “including but not limited to,” unless otherwise specifically emphasized.
[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0174] In the embodiments provided in the present application, it should be understood that the disclosed devices, computer equipment and methods can be implemented in other ways. For example, the device and computer equipment embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0175] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data leakage protection method, characterized in that: include: Obtain the data stream to be detected in real time; Sending the data stream to be detected to the first detection system and the distributed message processing cluster in parallel; wherein the distributed message processing cluster is configured with a subscription relationship with the second detection system, and the second detection system is used to extract the data stream to be detected received by the distributed message processing cluster based on the subscription relationship; Using the first detection system and / or the second detection system to perform real-time data leakage risk detection on the data stream to be detected in parallel; When either the first detection system or the second detection system detects that the data stream to be detected has a risk of data leakage, a data blocking operation is performed.
2. The method according to claim 1, characterized in that The first detection system is a network application firewall, the second detection system is a data risk identification system, and the use of the first detection system and / or the second detection system to perform real-time data leakage risk detection on the data stream to be detected in parallel includes: Using the network application firewall to perform real-time data leakage risk detection on the data flow to be detected, to obtain a first detection result; The data risk identification system is used to perform real-time data leakage risk detection on the data flow to be detected according to preset risk identification rules to obtain a second detection result. The preset risk identification rules include at least one of the following: sensitive data identification rules in a set regular manner, access request frequency analysis rules, and access volume and access time analysis rules.
3. The method according to claim 2, characterized in that The first detection system includes a first risk identification device, the second detection system includes a second risk identification device, and the data risk identification system is used to perform real-time data leakage risk detection on the data flow to be detected according to a preset risk identification rule to obtain a second detection result, including: Determine the virtual node number of the second risk identification device by using a consistent hash algorithm; Performing a hash calculation on the serial number name of the first risk identification device and the address name of the user terminal to obtain an association table, wherein the association table is used to record the association relationship between the first risk identification device, the second risk identification device and the user terminal, and the user terminal is used to indicate the source of the data flow to be detected; The data stream to be detected is distributed to the virtual node of the corresponding second risk identification device according to the association table, and real-time data leakage risk detection is performed to obtain the second detection result.
4. The method according to claim 2, characterized in that: The using the data risk identification system to perform real-time data leakage risk detection on the to-be-detected data stream according to preset risk identification rules to obtain a second detection result includes: Retrieving historical detection data streams from the data risk identification system; The historical detection data stream and the data stream to be detected are analyzed in parallel according to the preset risk identification rule to obtain the second detection result.
5. The method according to claim 1, characterized in that: The real-time acquisition of the data stream to be detected includes: Configure a preset data acquisition rule, wherein the preset data acquisition rule includes at least one of the following: target name, target port, and target protocol; According to the preset data acquisition rules, the network application firewall is used to obtain the data flow to be detected in real time.
6. The method according to any one of claims 1 to 5, characterized in that When any one of the first detection system or the second detection system detects that the data stream to be detected has a data leakage risk, performing a data blocking operation includes: When the first detection result has a data leakage risk, the network connection of the user terminal address having the data leakage risk is terminated by using a network application firewall.
7. The method according to claim 6, characterized in that The method further comprises: When there is a risk of data leakage in the second detection result, based on the subscription relationship configured between the second detection system and the distributed message processing cluster, the second detection system sends a warning notification to the distributed message processing cluster, and the distributed message processing cluster is used to instruct the network application firewall to terminate the network connection of the user-side address with the risk of data leakage according to the warning notification.
8. A data leakage protection device, characterized in that: include: An acquisition unit, used for acquiring the data stream to be detected in real time; A processing unit, configured to send the data stream to be detected to the first detection system and the distributed message processing cluster in parallel; wherein the distributed message processing cluster is configured with a subscription relationship with the second detection system, and the second detection system is configured to extract the data stream to be detected received by the distributed message processing cluster based on the subscription relationship; The processing unit is further used to perform real-time data leakage risk detection on the data stream to be detected in parallel using the first detection system and / or the second detection system; The processing unit is further configured to perform a data blocking operation when either the first detection system or the second detection system detects that the data stream to be detected has a risk of data leakage.
9. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer device, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Sensitive information detection method and device, storage medium and computer device
CN110598411A
Traffic data detection method and system, storage medium and electronic device
CN111404890A
Pervasive, domain and situational-aware, adaptive, automated, and coordinated analysis and control of enterprise-wide computers, networks, and applications for mitigation of business and operational risks and enhancement of cyber security
US20130104236A1
Distributed security testing system, method and device, and storage medium
WO2021097713A1
Network security risk inspection method and system, computer device, and storage medium
WO2021174870A1
