Enterprise data security management method and system suitable for big data analysis

Abstract

The invention discloses an enterprise data security management method and system suitable for big data analysis, particularly relates to the technical field of data processing related to computing systems or devices, and comprises a block chain technology, hash operation, authority control, an adaptive segmentation algorithm and the like. The source, the flow direction and the transmission time of each data transmission are recorded through the private block chain, a non-tampering record is generated, even if the data frequently flows among multiple departments, the transmission path of the data can be tracked, and the security and the integrity of the data are ensured; in addition, the number and granularity of data segments are dynamically adjusted according to the data transmission condition and the evaluation result, the safety management efficiency in a complex data flow environment is further improved, and the complexity of enterprise data management is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology related to computing systems or devices, and more specifically, to an enterprise data security management method and system suitable for big data analysis. Background Art

[0002] Enterprise data refers to various information generated and stored in enterprise operations, such as financial, customer, sales and other data. Existing technologies manage data through tools such as databases and cloud storage, and use technologies such as big data and artificial intelligence to extract valuable insights from it. At the same time, security technologies such as encryption and identity authentication can be used to protect the confidentiality and integrity of data and prevent leakage and abuse;

[0003] In an enterprise, some data needs to flow frequently between departments, which makes the security management of this data extremely inconvenient. Due to the frequent transmission of data between departments, the management difficulty is greatly increased. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an enterprise data security management method and system suitable for big data analysis, which dynamically adjusts the number and granularity of data segments according to the data transmission situation and evaluation results to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: an enterprise data security management method suitable for big data analysis, comprising:

[0006] Establish a private blockchain network in all departments within the enterprise. Before each data flow, the source information, flow information, and transmission time information of the data are written into the blockchain through the private blockchain network to generate an unalterable record. Use the Merkle tree algorithm to perform hash operations to detect whether the data has been tampered with. Alerts are issued and marked for tampered data.

[0007] Analyze the flow status of untampered data and divide the data into abnormal flow data and normal flow data based on the transmission path deviation ratio and permission usage rate; when the transmission path deviation ratio is less than the deviation ratio threshold and the permission usage rate is less than the usage rate threshold, the data is regarded as normal flow data; when the transmission path deviation ratio is greater than or equal to the deviation ratio threshold or the permission usage rate is greater than or equal to the usage rate threshold, the data is regarded as abnormal flow data;

[0008] When data is classified as abnormal flow data, the data is marked and isolated from the normal transmission queue and stored in a protected isolation area;

[0009] Through an adaptive algorithm, each abnormal flow data is divided into N independent paragraphs to form a set of the abnormal flow data. Independent identifiers and metadata are marked for each independent paragraph in the set, and data transmission integrity and hash consistency are evaluated for the N independent paragraphs in turn.

[0010] When the data transmission integrity is less than the integrity threshold and the hash consistency evaluation passes, the system classifies the data segment as a normal segment; when the data transmission integrity is greater than or equal to the integrity threshold or the hash consistency evaluation fails, the system classifies the data segment as an abnormal segment;

[0011] After evaluation, normal paragraphs are moved from the isolation area back to the normal transmission queue to restore the flow state. For abnormal paragraphs, the system continues to isolate them and triggers a manual review signal.

[0012] The value of N is dynamically adjusted based on the return rate of independent paragraphs.

[0013] In a preferred embodiment, during the data transmission process, a root hash value H is generated by constructing a Merkle tree. Merkle , used to verify the integrity of data transmission; first, the source information D of the data source , data flow information D destination And data transmission time information T transmission Hash the nodes separately to form leaf nodes. Then merge the leaf nodes in pairs and calculate the hash values ​​layer by layer to finally generate the root hash value H. Merkle ;

[0014] H Merkle =H(H Leaf1 ||H Leaf2 );

[0015] H Leaf1 =H(D source )||H(D destination )and H Leaf2 =H(T transmission )||H(D other );

[0016] Among them, H Merkle Represents the root hash value generated by the Merkle tree algorithm, which is used to verify the integrity of data transmission; H Leaf1 and H Leaf2 Represents the hash value of the leaf node of the Merkle tree, which calculates the source information of the data, the flow information of the data, and the transmission time information of the data; D source Indicates the source information of the data, that is, the sender of the data; D destination Indicates the destination information of the data, that is, the receiver; T transmissionIndicates the transmission time of the data, that is, the timestamp of the transmission; D other Represents other metadata; H(X is the hash function SHA-256, which is used to generate a 256-bit hash value; by recursively calculating the hash values ​​of these leaf nodes, the root hash value H is finally generated. Merkle Used to verify whether tampering has occurred during transmission.

[0017] In a preferred embodiment, during data transmission, the system not only calculates the transmission path deviation ratio and the authority usage rate, but also dynamically adjusts the predetermined path L expected and permission A total By analyzing network load and permission usage in real time, these baseline values ​​can be adjusted to cope with dynamic changes in different network environments and permission usage;

[0018]

[0019] Where P deviation It represents the transmission path deviation ratio, that is, the deviation ratio between the actual path and the expected path; L actual Indicates the actual path length during data transmission, that is, the number of nodes actually passed; L expected It indicates the expected path length adjusted in real time according to the dynamic network load, reflecting the optimal path of the network status during transmission; U permission Indicates the usage rate of permissions in the current transmission, that is, the proportion of permissions used; A current Indicates the number of permissions currently used; A total Indicates the total number of permissions allocated that the system dynamically adjusts for the transmission; detects anomalies in data transmission by calculating the path deviation ratio and permission usage rate, and makes adjustments based on changes in the network and permission load.

[0020] In a preferred embodiment, for abnormal data D anomaly , the system uses an adaptive algorithm to segment it; each data segment d i The segmentation is dynamically adjusted according to its content, data size, and transmission conditions, and a unique identifier and metadata are assigned to each segment. The adaptive algorithm determines the reasonable number of segments N by analyzing the complexity of the data;

[0021] D segment ={d 1 ,d 2 ,…,d N where d i =Segment(D anomaly ,N)

[0022] Where D segment Represents the set of paragraphs after the abnormal data is segmented; d irepresents the i-th data segment, each data segment contains an independent identifier and metadata; D anomaly represents the complete data detected as anomalies; N represents the number of paragraphs determined by the adaptive algorithm.

[0023] In a preferred embodiment, the system performs a i Conduct assessments, including transmission integrity I transmission and hash consistency H consistent ,Transmission integrity is calculated by the ratio of error rate to total transmission times, while hash consistency confirms data consistency by comparing the current hash value of the data segment with the original hash value;

[0024]

[0025] Among them I transmission Indicates the transmission integrity of each data segment, calculating the proportion of errors in the total transmission; E error Indicates the number of errors in the data transmission process; E total Indicates the total number of transmissions of the data segment; H consistent Indicates hash consistency, which is used to check whether the data has been tampered with by comparing the hash values. i Represents the current hash value of the i-th segment; H original (d i ) represents the original hash value of the i-th segment.

[0026] In a preferred embodiment, after processing the data segment, the system will calculate the return rate R return The number of segments N is adjusted dynamically, and the return rate is determined by the ratio of the number of data segments marked as normal after evaluation to the total number of segments;

[0027]

[0028] Where R return It represents the return rate of normal paragraphs after evaluation, that is, the percentage of normal paragraphs in the total number of paragraphs; S normal Indicates the number of paragraphs marked as normal; N indicates the current number of segments; N new represents the number of new segments adjusted according to the return rate; α represents the increase coefficient when the return rate is higher than the threshold; β represents the reduction coefficient when the return rate is lower than the threshold; R threshold Indicates the return rate threshold set by the adaptive algorithm.

[0029] In a preferred embodiment, an enterprise data security management system suitable for big data analysis includes a blockchain recording module, a hash detection module, a data flow analysis module, a data isolation module, an adaptive segmentation and evaluation module, a normal and abnormal data processing module, and a dynamic adjustment module;

[0030] The blockchain record module is used to establish a private blockchain network within the enterprise. Every time data flows, the source information, flow information, and transmission time information of the data are written into the blockchain to generate an unalterable record.

[0031] The hash detection module uses the Merkle tree algorithm to perform hash operations on the data to detect whether the data has been tampered with during transmission. The hash detection module calculates the hash value in real time during data transmission and compares the result with the expected hash value. For tampered data, an alarm is issued and marked.

[0032] The data flow analysis module is used to analyze the flow status of untampered data. By calculating the transmission path deviation ratio and permission usage rate, the data is divided into abnormal flow data and normal flow data, and marked accordingly.

[0033] The data isolation module isolates data from the normal transmission queue and stores it in a protected isolation area when the data is classified as abnormal flow data;

[0034] The adaptive segmentation and evaluation module divides each abnormal data into multiple independent segments through an adaptive algorithm. Each segment has an independent identifier and metadata, and evaluates its data transmission integrity and hash consistency segment by segment.

[0035] Normal and abnormal data processing module, after evaluating each segment, normal segments will be moved back to the normal transmission queue from the isolation area to resume data flow; while abnormal segments will continue to be isolated and trigger manual review signals for further processing;

[0036] The dynamic adjustment module adjusts the number of segments based on the return rate of independent segments during the evaluation process.

[0037] Technical effects and advantages of the present invention:

[0038] 1. The private blockchain records the source, direction and time of each data transmission, generating tamper-proof records. Even if data frequently flows between multiple departments, its transmission path can be tracked to ensure data security and integrity. In addition, the number and granularity of data segments are dynamically adjusted according to the data transmission situation and evaluation results, further improving the security management efficiency in a complex data flow environment and reducing the complexity of enterprise data management.

[0039] 2. By performing hash operations on data through the Merkle tree algorithm, the system can quickly detect any data tampering during transmission and immediately issue an alarm to ensure the authenticity and security of data during transmission;

[0040] 3. By calculating the data transmission path deviation ratio and permission usage rate in real time, unauthorized access or transmission paths that do not meet expectations can be identified in a timely manner, thereby isolating potential security threats in data flow and preventing data leakage or abuse;

[0041] 4. Efficient and flexible data processing is achieved through adaptive data segmentation and dynamic adjustment of the number of segments based on the return rate. The segmentation strategy can be dynamically optimized based on the actual results of data evaluation, thereby improving the flexibility and processing efficiency of data security management. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Refer to the instruction manual Figure 1 , an enterprise data security management method applicable to big data analysis according to an embodiment of the present invention comprises:

[0045] Establish a private blockchain network in all departments within the enterprise. Before each data flow, the source information, flow information, and transmission time information of the data are written into the blockchain through the private blockchain network to generate an unalterable record. Use the Merkle tree algorithm to perform hash operations to detect whether the data has been tampered with. Alerts are issued and marked for tampered data.

[0046] Analyze the flow status of untampered data and divide the data into abnormal flow data and normal flow data based on the transmission path deviation ratio and permission usage rate; when the transmission path deviation ratio is less than the deviation ratio threshold and the permission usage rate is less than the usage rate threshold, the data is regarded as normal flow data; when the transmission path deviation ratio is greater than or equal to the deviation ratio threshold or the permission usage rate is greater than or equal to the usage rate threshold, the data is regarded as abnormal flow data;

[0047] When data is classified as abnormal flow data, the data is marked and isolated from the normal transmission queue and stored in a protected isolation area;

[0048] Through an adaptive algorithm, each abnormal flow data is divided into N independent paragraphs to form a set of the abnormal flow data. Independent identifiers and metadata are marked for each independent paragraph in the set, and data transmission integrity and hash consistency are evaluated for the N independent paragraphs in turn.

[0049] When the data transmission integrity is less than the integrity threshold and the hash consistency evaluation passes, the system classifies the data segment as a normal segment; when the data transmission integrity is greater than or equal to the integrity threshold or the hash consistency evaluation fails, the system classifies the data segment as an abnormal segment;

[0050] After evaluation, normal paragraphs are moved from the isolation area back to the normal transmission queue to restore the flow state. For abnormal paragraphs, the system continues to isolate them and triggers a manual review signal.

[0051] The value of N is dynamically adjusted based on the return rate of independent paragraphs.

[0052] Furthermore, during the data transmission process, the root hash value H is generated by constructing a Merkle tree. Merkle , used to verify the integrity of data transmission; first, the source information D of the data source , data flow information D destination And data transmission time information T transmission Hash the nodes separately to form leaf nodes. Then merge the leaf nodes in pairs and calculate the hash values ​​layer by layer to finally generate the root hash value H. Merkle ; In this way, even if part of the data segment is tampered with, the root hash value of the entire Merkle tree will change, thereby detecting data tampering;

[0053] H Merkle =H(H Leaf1 ||H Leaf2 );

[0054] H Leaf1 =H(D source )||H(D destination )and H Leaf2 =H(T transmission )||

[0055] H(D other );

[0056] Among them, H Merkle Represents the root hash value generated by the Merkle tree algorithm, which is used to verify the integrity of data transmission; H Leaf1 and H Leaf2 Represents the hash value of the leaf node of the Merkle tree, which calculates the source information of the data, the flow information of the data, and the transmission time information of the data; D source Indicates the source information of the data, that is, the sender of the data; D destination Indicates the destination information of the data, that is, the receiver; T transmission Indicates the transmission time of the data, that is, the timestamp of the transmission; D otherRepresents possible other metadata, such as security identifiers, etc.; H(X) is the hash function SHA-256, which is used to generate a 256-bit hash value; by recursively calculating the hash values ​​of these leaf nodes, the root hash value H is finally generated Merkle Used to verify whether tampering has occurred during transmission.

[0057] Furthermore, during data transmission, the system not only calculates the transmission path deviation ratio and the authority usage rate, but also dynamically adjusts the scheduled path L expected and permission A total By analyzing network load and permission usage in real time, these baseline values ​​can be adjusted to cope with dynamic changes in different network environments and permission usage;

[0058]

[0059] Where P deviation It represents the transmission path deviation ratio, that is, the deviation ratio between the actual path and the expected path; L actual Indicates the actual path length during data transmission, that is, the number of nodes actually passed; L expected It indicates the expected path length adjusted in real time according to the dynamic network load, reflecting the optimal path of the network status during transmission; U permission Indicates the usage rate of permissions in the current transmission, that is, the proportion of permissions used; A current Indicates the number of permissions currently in use, such as the user's permission access amount; A total Indicates the total number of permissions allocated for the transmission that the system dynamically adjusts based on the current system security policy; detects anomalies in data transmission by calculating the path deviation ratio and permission usage rate, and makes adjustments based on changes in the network and permission load.

[0060] Furthermore, for abnormal data D anomaly , the system uses an adaptive algorithm to segment it; each data segment d i The segmentation is dynamically adjusted according to its content, data size, transmission conditions, etc., and a unique identifier and metadata are assigned to each segment. The adaptive algorithm determines the reasonable number of segments N by analyzing the complexity of the data;

[0061] D segment ={d 1 ,d 2 ,…,d N where d i =Segment(D anomaly ,N)

[0062] Where D segment Represents the set of paragraphs after the abnormal data is segmented; d irepresents the i-th data segment, each data segment contains an independent identifier and metadata; D anomaly Represents the complete data detected as abnormal; N represents the number of paragraphs determined by the adaptive algorithm, which is adjusted by the system according to the data size and complexity; each data paragraph will be marked as an independent identifier and accompanied by relevant metadata (such as paragraph size, hash value, etc.) to ensure that the system can perform detailed evaluation on a paragraph by paragraph basis.

[0063] Furthermore, the system performs a i Conduct assessments, including transmission integrity I transmission and hash consistency H consistent ,Transmission integrity is calculated by the ratio of error rate to total transmission times, while hash consistency confirms data consistency by comparing the current hash value of the data segment with the original hash value;

[0064]

[0065] Among them I transmission Indicates the transmission integrity of each data segment, calculating the proportion of errors in the total transmission; E error Indicates the number of errors during data transmission, such as packet loss or network interruption; E total Indicates the total number of transmissions of the data segment, including all transmission packets; H consistent Indicates hash consistency, which is used to check whether the data has been tampered with by comparing the hash values. i ) represents the current hash value of the i-th segment; H original (d i ) represents the original hash value of the i-th paragraph; if the transmission integrity is lower than the set threshold and the hash consistency check passes, the paragraph is considered normal; otherwise, the paragraph is marked as abnormal for further processing.

[0066] Furthermore, after processing the data segment, the system will calculate the return rate R. return Dynamically adjust the number of segments N. The return rate is determined by the ratio of the number of data segments marked as normal after evaluation to the total number of segments. If the return rate is high, the system will increase the value of the segment number N to improve processing efficiency. If the return rate is low, the system will reduce the value of N to analyze the data more carefully.

[0067]

[0068] Where R return It represents the return rate of normal paragraphs after evaluation, that is, the percentage of normal paragraphs in the total number of paragraphs; S normal Indicates the number of paragraphs marked as normal; N indicates the current number of segments; N newrepresents the number of new segments adjusted according to the return rate; α represents the increase coefficient when the return rate is higher than the threshold; β represents the reduction coefficient when the return rate is lower than the threshold; R threshold Represents the return rate threshold set by the adaptive algorithm. By adjusting N, the system dynamically changes the granularity of the segmentation according to the return rate to ensure that data can be processed in the most efficient way in different scenarios.

[0069] An enterprise data security management system suitable for big data analysis, including a blockchain recording module, a hash detection module, a data flow analysis module, a data isolation module, an adaptive segmentation and evaluation module, a normal and abnormal data processing module, and a dynamic adjustment module;

[0070] The blockchain record module is used to establish a private blockchain network within the enterprise. Every time data flows, the source information, flow information, and transmission time information of the data are written into the blockchain to generate an unalterable record.

[0071] The hash detection module uses the Merkle tree algorithm to perform hash operations on the data to detect whether the data has been tampered with during transmission. The hash detection module calculates the hash value in real time during data transmission and compares the result with the expected hash value. For tampered data, an alarm is issued and marked.

[0072] The data flow analysis module is used to analyze the flow status of untampered data. By calculating the transmission path deviation ratio and permission usage rate, the data is divided into abnormal flow data and normal flow data, and marked accordingly.

[0073] The data isolation module isolates data from the normal transmission queue and stores it in a protected isolation area when the data is classified as abnormal flow data;

[0074] The adaptive segmentation and evaluation module divides each abnormal data into multiple independent segments through an adaptive algorithm. Each segment has an independent identifier and metadata, and evaluates its data transmission integrity and hash consistency segment by segment.

[0075] Normal and abnormal data processing module, after evaluating each segment, normal segments will be moved back to the normal transmission queue from the isolation area to resume data flow; while abnormal segments will continue to be isolated and trigger manual review signals for further processing;

[0076] The dynamic adjustment module adjusts the number of segments based on the return rate of independent segments during the evaluation process, ensuring that abnormal data can be accurately identified and processed while processing data efficiently.

[0077] It should be noted that: in this solution, the enterprise uses private blockchain technology to ensure the secure flow of data; before each data flow, the system will record the source information, destination information and transmission time of the data in the blockchain; this information is stored as an unalterable record on the blockchain to ensure traceability and tamper-proofness throughout the data transmission process; the private blockchain is essentially a decentralized distributed ledger, and each node in the system has an identical copy. When data is written to the chain, it is recorded in each copy to ensure that all departments can access and verify the data flow record at the same time; the implementation of this mechanism ensures that even in big data analysis, the flow of data is always transparent and secure; the blockchain uses encryption and distributed structure to ensure that even if a single node is hacked, the integrity of the data will not be threatened;

[0078] Next, the system uses an algorithm called Merkle tree to perform hash operations to detect whether the data has been tampered with during transmission. The Merkle tree is a tree-like data structure. Each node at the bottom layer represents the hash value of a data block. The hash values ​​of two adjacent nodes are combined to generate a higher-level hash value, and finally a root hash value is formed at the top layer. When data is transmitted, the system will hash the source information, destination information and transmission time of the data, and this information constitutes the leaf nodes of the Merkle tree. If any part of the data changes during the transmission process, the top-level root hash value of the Merkle tree will change accordingly, so that the system can immediately detect data tampering. Compared with ordinary hash algorithms, the Merkle tree has the characteristics of recursion and efficiency. It can handle massive amounts of data and provides higher security and transparency in structure.

[0079] During the data flow process, the system further evaluates whether the data flows according to the expected path and permissions by calculating the transmission path deviation ratio and permission usage rate; the transmission path deviation ratio refers to the difference between the actual data transmission path and the predetermined path. The system calculates the percentage of deviation by monitoring the data flow of each node in real time; if the deviation exceeds a certain set threshold, an alarm will be triggered, indicating that the data may not flow according to the path specified by security regulations; similarly, the permission usage rate refers to the ratio between the permissions actually used during the transmission process and the number of pre-allocated permissions; by detecting the permission usage, the system can determine whether there is unauthorized access or abuse of permissions; if the permission usage rate is found to be abnormally high or exceeds the preset range, the system will immediately mark the data as abnormal;

[0080] To handle abnormal data, the system uses an adaptive algorithm. When data is marked as abnormal, the system does not process the entire data block at once, but divides it into multiple independent segments. Each segment is divided according to data size, complexity, and transmission conditions, and the system assigns a unique identifier and related metadata to each segment. These metadata include the size of the segment, hash value, and transmission history. The adaptive algorithm dynamically adjusts the number of segments by analyzing the return rate of data segments. If the data is confirmed to be normal after evaluation, the segments will be returned to the normal transmission queue. If there are still problems with the data, the system will continue to isolate it and trigger a signal for manual review. The adaptive algorithm dynamically adjusts the number of segments to ensure that it can maintain high efficiency and provide detailed anomaly detection when handling abnormal data.

[0081] In addition, the specific implementation of the adaptive algorithm in this solution is: the system dynamically adjusts the data segmentation strategy according to the data processing results through real-time monitoring and feedback mechanisms; first, when abnormal data is detected, the system does not use a fixed number of segments, but calculates the return rate through the evaluation results of each data segment, that is, the proportion of data segments marked as normal to the total data segments; assuming that a certain data is divided into several segments, each segment undergoes an independent integrity check and hash consistency evaluation. If most of the segments are evaluated as normal, the system will consider that the data problem is relatively minor and increase the granularity of data segmentation, that is, increase the number of segments, thereby speeding up subsequent processing; if the return rate is low, indicating that the data problems are more or more complex, the system will reduce the number of segments, thereby performing a more detailed segment-by-segment analysis of the data;

[0082] Through this feedback mechanism, the system can continuously adjust its behavior according to the actual processing results to adapt to the complexity and abnormality of different data. Specifically, the system will monitor the evaluation results in real time, such as the transmission integrity and hash consistency of each data segment. If these evaluation indicators show that most data segments are normal, the system will increase the granularity of segmentation in subsequent data processing, that is, use more smaller segments for parallel processing to improve overall processing efficiency. If the evaluation shows that most segments have problems, the system will choose to reduce the number of segments in order to more deeply examine the details of each segment. This adaptive adjustment mechanism ensures that the system can flexibly respond to data problems of different types and sizes, so that the system's processing can maintain high efficiency and fully detect potential anomalies.

[0083] In addition, if the data structure is different, the system can adopt different segmentation strategies according to the type, content, size and complexity of the specific data, so as to more effectively process different types of data; the following details the processing methods of various different data structures;

[0084] First, for structured data, such as common database tables or Excel files, this type of data usually has a clear row and column structure; in this case, the system can segment according to rows or columns; for example, suppose there is a database table containing 10,000 user data, each row represents the detailed information of a user; the system can select every 500 rows as a data segment and divide the entire table into 20 segments; if most of the segments are considered normal after data evaluation, the system can improve processing efficiency and expand each segment to 1,000 rows and divide it into 10 segments; if many abnormal segments are found, the system will reduce the number of segmented rows, such as reducing each segment to 200 rows, so as to analyze each segment in a more granular manner; this record-based segmentation method ensures that the system can flexibly adjust the processing granularity according to the amount of data, especially when processing massive data.

[0085] Next, for semi-structured data, such as JSON and XML files, although the data has a certain hierarchical structure, the content and size of each data node may be inconsistent. When processing such data, the system usually segments it according to nodes or fields. For example, suppose a JSON file contains order data of multiple users, and each order has a different number of commodity items. The system can segment it according to each user or each order. If the evaluation finds that most users or order segments are normal, the system may select multiple user data or multiple orders as one data segment, thereby reducing the number of segments. If some order data is found to be abnormal, the system will reduce the segmentation granularity and may segment it according to each commodity item to analyze each part of the order in more detail. In this way, the system can adaptively adjust the segmentation strategy to process semi-structured data at different levels.

[0086] For unstructured data, such as text, images, audio, or video, the segmentation strategy is more flexible and is usually based on the size, content, and specific processing requirements of the data. For example, for text data, suppose the system is processing a document containing 10,000 characters. The system can first segment by natural paragraphs, assuming that the text has 50 paragraphs, each with about 200 characters. If the evaluation finds that most paragraphs are normal, the system may expand the number of characters in the paragraph to 500 and reduce the number of segmentations to improve processing efficiency. If multiple paragraphs are detected to have problems, the system will reduce the segmentation granularity and even segment them by each sentence to analyze the anomalies in the text in more detail. For image data, the system may segment according to the characteristics of the image size or content, such as dividing a high-resolution image into multiple small blocks by pixel area for processing. If the evaluation finds that most image blocks are normal, the system can increase the size of the image blocks to reduce processing time. If there are multiple abnormal blocks, the system will refine the block size to analyze the problems in the image one by one.

[0087] The processing of audio or video data can be based on the timeline or the number of frames. For example, if a video of one hour is to be processed, the system can divide it into six segments of 10 minutes each for processing. If most of the video segments are evaluated as normal, the system can increase the segmentation granularity, for example, divide it into four segments of 15 minutes each. If multiple video segments are found to be abnormal after evaluation, the system may divide the video into segments of 5 minutes or even 1 minute each, so as to examine each part of the video in more detail. This segmentation method based on the timeline or the number of frames can flexibly adjust the segment size when processing multimedia data, ensuring accurate detection and processing of abnormalities.

[0088] Finally, for continuous data streams such as log files or streaming data, the system usually segments them according to time windows or events. For example, when processing a network log, the system can process each hour of log data as a segment. If the log volume is very large, the system can split the log per minute to form more small segments for refined processing. For streaming data, the system may segment according to the number of events or packet size in the data stream, and adjust the number and size of segments in real time. If the evaluation finds that most of the packets are normal, the system will increase the number of packets in each segment to improve the processing speed. If an abnormality is found, the system will reduce the number of packets in each segment to analyze the specific problems of each packet in depth.

[0089] Therefore, when the system processes different data structures, it can refer to different segmentation strategies according to specific situations; structured data can be segmented by rows or columns, semi-structured data can be segmented by nodes or fields, unstructured data can be segmented by natural content (such as text paragraphs, image areas or video timelines), and continuous data streams can be segmented by time windows or events; according to the processing results, the size and number of segments are dynamically adjusted through adaptive algorithms to ensure both efficient processing and full detection of potential anomalies;

[0090] The system also implements an isolation mechanism. When data is detected as abnormal, the system will isolate it from the normal transmission queue to ensure that it does not affect the normal transmission of other data. The isolated data will be stored in a protected area until its problem is resolved. During this period, the system will regularly evaluate the isolated data to detect its transmission integrity and hash consistency. Transmission integrity refers to whether the data is lost or whether a transmission error occurs during the transmission process, while hash consistency determines whether the data has been tampered with by comparing the hash value of the current data segment with the original hash value. Only when the data passes these two evaluations will the system put it back into the normal transmission process. If the evaluation fails, the data will continue to remain in the isolation area and trigger manual intervention as needed.

[0091] In addition, the dynamic adjustment module in the system determines whether to increase or decrease the number of data segments by detecting the return rate of independent paragraphs; the return rate refers to the proportion of data segments marked as normal after evaluation to the total number of paragraphs; if the return rate is high, the system will automatically increase the number of segments to improve the efficiency of data processing; if the return rate is low, the system will reduce the number of segments to detect data transmission problems more carefully; through this adaptive mechanism, the system can make real-time adjustments according to different data situations, ensuring that it can process data efficiently and accurately detect and solve potential problems of abnormal data;

[0092] In general, this solution combines private blockchain, Merkle tree algorithm, adaptive segmentation processing and permission usage monitoring technologies to form a set of enterprise data security management methods and systems suitable for big data analysis; this solution can not only effectively prevent data tampering, but also monitor the flow status of data in real time to ensure the security and integrity of data during transmission; at the same time, through isolation and dynamic adjustment mechanisms, the system can carefully process abnormal data, reduce potential security risks, and provide efficient and flexible solutions; this design can adapt to the security needs of different enterprises, especially in the big data analysis environment, providing strong protection for enterprise data security;

[0093] In addition, this solution involves data processing methods for specific functions, which comply with the G06F17 classification number, especially in the field of enterprise data security management; the solution ensures the security and integrity of big data through steps such as private blockchain records, Merkle tree algorithm to detect tampering, transmission path deviation ratio and permission usage rate analysis, and adaptive algorithm for data segmentation and dynamic adjustment; these technical operations are all for the purpose of realizing specialized data processing functions, especially data transmission, verification and exception handling, which comply with the description of data processing methods for specific functions in G06F17.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An enterprise data security management method suitable for big data analysis, characterized in that: include: Establish a private blockchain network in all departments within the enterprise. Before each data flow, the source information, flow information, and transmission time information of the data are written into the blockchain through the private blockchain network to generate an unalterable record. Use the Merkle tree algorithm to perform hash operations to detect whether the data has been tampered with. Alerts are issued and marked for tampered data. Analyze the flow status of untampered data and divide the data into abnormal flow data and normal flow data based on the transmission path deviation ratio and permission usage rate; When the transmission path deviation ratio is less than the deviation ratio threshold and the permission usage rate is less than the usage rate threshold, the data is considered normal flow data; when the transmission path deviation ratio is greater than or equal to the deviation ratio threshold or the permission usage rate is greater than or equal to the usage rate threshold, the data is considered abnormal flow data; When data is classified as abnormal flow data, the data is marked and isolated from the normal transmission queue and stored in a protected isolation area; Through an adaptive algorithm, each abnormal flow data is divided into N independent paragraphs to form a set of the abnormal flow data. Independent identifiers and metadata are marked for each independent paragraph in the set, and data transmission integrity and hash consistency are evaluated for the N independent paragraphs in turn. When the data transmission integrity is less than the integrity threshold and the hash consistency evaluation passes, the system classifies the data segment as a normal segment; when the data transmission integrity is greater than or equal to the integrity threshold or the hash consistency evaluation fails, the system classifies the data segment as an abnormal segment; After evaluation, normal paragraphs are moved from the isolation area back to the normal transmission queue to restore the flow state. For abnormal paragraphs, the system continues to isolate them and triggers a manual review signal. The value of N is dynamically adjusted based on the return rate of independent paragraphs.

2. The enterprise data security management method suitable for big data analysis according to claim 1, characterized in that: During the data transmission process, the root hash value H is generated by constructing a Merkle tree. Merkle , used to verify the integrity of data transmission; first, the source information D of the data source , data flow information D destination And data transmission time information T transmission Hash the nodes separately to form leaf nodes. Then merge the leaf nodes in pairs and calculate the hash values ​​layer by layer to finally generate the root hash value H. Merkle ; H Merkle =H(H Leaf1 ∥H Leaf2 ); H Leaf1 =H(D source )|||H(D destination )and H Leaf2 =H(T transmission )|||H(D other ); Among them, H Merkle Represents the root hash value generated by the Merkle tree algorithm, which is used to verify the integrity of data transmission; H Leaf1 and H Leaf2 Represents the hash value of the leaf node of the Merkle tree, which calculates the source information of the data, the flow information of the data, and the transmission time information of the data; D source Indicates the source information of the data, that is, the sender of the data; D destination Indicates the destination information of the data, that is, the receiver; T transmission Indicates the transmission time of the data, that is, the timestamp of the transmission; D other Represents other metadata; H(X is the hash function SHA-256, which is used to generate a 256-bit hash value; by recursively calculating the hash values ​​of these leaf nodes, the root hash value H is finally generated. Merkle Used to verify whether tampering has occurred during transmission.

3. The enterprise data security management method suitable for big data analysis according to claim 2 is characterized by: During data transmission, the system not only calculates the transmission path deviation ratio and permission usage rate, but also dynamically adjusts the scheduled path L expected and permission A total By analyzing network load and permission usage in real time, these baseline values ​​can be adjusted to cope with dynamic changes in different network environments and permission usage; Where P deviation It represents the transmission path deviation ratio, that is, the deviation ratio between the actual path and the expected path; L actual Indicates the actual path length during data transmission, that is, the number of nodes actually passed; L expected It indicates the expected path length adjusted in real time according to the dynamic network load, reflecting the optimal path of the network status during transmission; U permission Indicates the usage rate of permissions in the current transmission, that is, the proportion of permissions used; A current Indicates the number of permissions currently used; A total Indicates the total number of permissions allocated for the transfer that the system dynamically adjusts; By calculating the path deviation ratio and permission usage rate, anomalies in data transmission are detected and adjustments are made based on changes in network and permission loads.

4. The enterprise data security management method suitable for big data analysis according to claim 3 is characterized by: For abnormal data D anomaly , the system uses an adaptive algorithm to segment it; each data segment d i The segmentation is dynamically adjusted according to its content, data size, and transmission conditions, and a unique identifier and metadata are assigned to each segment. The adaptive algorithm determines the reasonable number of segments N by analyzing the complexity of the data; D segment ={d1,d2,…,d N }where d i =Segment(D anomaly ,N) Where D segment Represents the set of paragraphs after the abnormal data is segmented; d i represents the i-th data segment, each data segment contains an independent identifier and metadata; D anomaly represents the complete data detected as anomalies; N represents the number of paragraphs determined by the adaptive algorithm.

5. The enterprise data security management method suitable for big data analysis according to claim 4 is characterized by: The system performs the following operations on each data segment: i Conduct assessments, including transmission integrity I transmission and hash consistency H consistent ,Transmission integrity is calculated by the ratio of error rate to total transmission times, while hash consistency confirms data consistency by comparing the current hash value of the data segment with the original hash value; Among them I transmission Indicates the transmission integrity of each data segment, calculating the proportion of errors in the total transmission; E error Indicates the number of errors in the data transmission process; E total Indicates the total number of transmissions of the data segment; H consistent Indicates hash consistency, which is used to check whether the data has been tampered with by comparing the hash values. i Represents the current hash value of the i-th segment; H original (d i ) represents the original hash value of the i-th segment.

6. The enterprise data security management method suitable for big data analysis according to claim 5, characterized in that: After processing the data segment, the system will return the data according to the return rate R. return The number of segments N is adjusted dynamically, and the return rate is determined by the ratio of the number of data segments marked as normal after evaluation to the total number of segments; Where R return It represents the return rate of normal paragraphs after evaluation, that is, the percentage of normal paragraphs in the total number of paragraphs; S normal Indicates the number of paragraphs marked as normal; N indicates the current number of segments; N new represents the number of new segments adjusted according to the return rate; α represents the increase coefficient when the return rate is higher than the threshold; β represents the reduction coefficient when the return rate is lower than the threshold; R threshold Indicates the return rate threshold set by the adaptive algorithm.

7. The enterprise data security management method applicable to big data analysis according to claim 6 further comprises an enterprise data security management system applicable to big data analysis, characterized in that: It includes blockchain recording module, hash detection module, data flow analysis module, data isolation module, adaptive segmentation and evaluation module, normal and abnormal data processing module, and dynamic adjustment module; The blockchain record module is used to establish a private blockchain network within the enterprise. Every time data flows, the source information, flow information, and transmission time information of the data are written into the blockchain to generate an unalterable record. The hash detection module uses the Merkle tree algorithm to perform hash operations on the data to detect whether the data has been tampered with during transmission. The hash detection module calculates the hash value in real time during data transmission and compares the result with the expected hash value. For tampered data, an alarm is issued and marked. The data flow analysis module is used to analyze the flow status of untampered data. By calculating the transmission path deviation ratio and permission usage rate, the data is divided into abnormal flow data and normal flow data, and marked accordingly. The data isolation module isolates data from the normal transmission queue and stores it in a protected isolation area when the data is classified as abnormal flow data; The adaptive segmentation and evaluation module divides each abnormal data into multiple independent segments through an adaptive algorithm. Each segment has an independent identifier and metadata, and evaluates its data transmission integrity and hash consistency segment by segment. Normal and abnormal data processing module, after evaluating each segment, normal segments will be moved back to the normal transmission queue from the isolation area to resume data flow; while abnormal segments will continue to be isolated and trigger manual review signals for further processing; The dynamic adjustment module adjusts the number of segments based on the return rate of independent segments during the evaluation process.

Images