A big data security storage method based on abnormal warning

Abstract

The present invention relates to the field of data security, and provides a big data security storage method based on abnormal warning, which obtains the activity data of the port by monitoring each port of the storage system, calculates the port deviation value according to the activity data of the port, locates the abnormal port by the port deviation value, performs security monitoring on the abnormal port and performs security verification on the data sent by it. The method can effectively improve the security and stability of the storage system, obtain and analyze the data of each port of the storage system in real time, accurately locate the port with potential abnormality or abnormality by using the port deviation value, and ensure the security of data transmission by timely monitoring and verification, fully reduce the data risk caused by potential security risks, improve the processing ability of the data storage system against malicious attacks, and effectively deal with various potential threats in the big data environment.

Description

Technical Field

[0001] The present invention relates to the field of data security, and in particular to a big data security storage method based on abnormal early warning. Background Art

[0002] With the development of the Internet and big data technology, the data storage environment has become increasingly complex, and the frequency and complexity of various malicious attacks have continued to increase. Chinese invention patent CN115333778B discloses a network attack behavior prediction method based on attack patterns. Specifically, it uses knowledge graph representation learning technology to achieve rapid and secure prediction of attack behaviors. However, in the field of big data storage, traditional security protection measures, such as firewalls, intrusion detection systems, and antivirus software, although they have certain effects in dealing with known threats, often appear to be relatively lagging in the face of new and complex real-time attacks, which can easily lead to data leakage, tampering and other problems, causing huge losses.

[0003] Modern malicious attacks are diverse and covert, such as distributed denial of service attacks, ransomware, phishing attacks, and advanced persistent threats (APTs). Attackers often use new technologies to bypass existing protection measures. Once the attack is successful, the integrity and security of the data will face great threats. Traditional protection methods are difficult to play an effective role in large-scale data storage environments due to their lack of flexibility and scalability.

[0004] Therefore, there is an urgent need for an innovative big data security storage solution that can detect and warn of malicious attacks in real time, automatically respond and provide effective protection in the early stages of attacks, so as to comprehensively improve the security and reliability of data storage. Summary of the invention

[0005] The purpose of the present invention is to propose a big data security storage method based on abnormal warning to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0006] The present invention provides a big data security storage method based on abnormal early warning, which obtains the activity data of the port by monitoring each port of the storage system, calculates the port deviation value according to the activity data of the port, locates the abnormal port by the port deviation value, performs security monitoring on the abnormal port and performs security verification on the data sent by it. The method can effectively improve the security and stability of the storage system, obtain and analyze the data of each port of the storage system in real time, accurately locate the port with potential abnormality or abnormality by using the port deviation value, and ensure the security of data transmission by timely monitoring and verification, fully reduce the data risk caused by potential security risks, improve the processing ability of the data storage system against malicious attacks, and effectively deal with various potential threats in the big data environment.

[0007] In order to achieve the above object, according to one aspect of the present invention, a big data security storage method based on abnormal early warning is provided, and the method comprises the following steps:

[0008] Monitor each port of the storage system and obtain the activity data of the port;

[0009] Calculate the port deviation value based on the port activity data, and locate the abnormal port through the port deviation value;

[0010] Perform security monitoring on abnormal ports and perform security verification on the data they send.

[0011] Furthermore, each port of the storage system refers to a data transmission port for data communication between the storage device and the storage system. After the storage device establishes a connection with the storage system through the data transmission port, data is sent from the storage device to the storage system to complete the data transmission; a single or multiple storage devices can establish a connection with the storage system through the same port.

[0012] Furthermore, the storage system is one of CubeFS, HDFS, and Ceph.

[0013] Furthermore, the method of monitoring each port of the storage system and obtaining the activity data of the port is as follows: let the total number of ports of the storage system be N, and let Port(i) be the i-th port among the N ports;

[0014] Set a monitoring period, record the bandwidth utilization of the port every T minutes during the monitoring period, and obtain the bandwidth utilization of each port at multiple times (one time is generated every T minutes, thus forming multiple times), and use array Bur(i) to store the bandwidth utilization of Port(i) at multiple times;

[0015] Then the N ports correspond to N arrays Bur(1), Bur(2),…, Bur(N), and these N arrays are used as active data.

[0016] Furthermore, the bandwidth utilization of a port refers to the instantaneous bandwidth utilization of the port at any moment, or refers to the average bandwidth utilization of the port in a time period formed between any moment and the moment before the moment (the length of the formed time period is the above-mentioned T minutes), and the average bandwidth utilization in the time period is defined as the average value of the instantaneous bandwidth utilization at all moments per second in the time period.

[0017] Preferably, the monitoring period is set to 24 hours or 48 hours, and T is set to 5 or 10.

[0018] Furthermore, the method for calculating the port bias value according to the activity data of the port is specifically as follows:

[0019] S1, for the i-th array Bur(i) in the N arrays Bur(1), Bur(2), ..., Bur(N) of activity data, MBur(i) represents the average value of all values ​​in Bur(i), and O i is the sum of all values ​​in the array Bur(i) that are greater than MBur(i), denoted by U i is the sum of all values ​​in the array Bur(i) that are less than MBur(i);

[0020] Filter out the array with the largest sum of all values ​​in the N arrays Bur(1), Bur(2), ..., Bur(N) and record it as Bur(P);

[0021] Set variable H0, and set the initial value of H0 to 0 P ÷U P , set variable j = 1, and use Bur(j) to represent the jth array corresponding to the value of variable j in N arrays Bur(1), Bur(2), …, Bur(N);

[0022] Definition j-1 =O P , U j-1 =U P (At this time O j-1 =O0=O P , U j-1 =U0=U P The purpose of this setting is to facilitate the subsequent update of the value of H0, because the value of H0 is only needed for the first time in the iteration. P and U P In the second and subsequent iterations, we need to rely on the O corresponding to Bur(j-1) in the previous round. j-1 and U j-1 ), go to S2;

[0023] S2, balance amount b j =|O j -U j |÷MAX{O j ,U j}, if H0≥1, then update the value of H0 to O j-1 ÷(U j-1 +b j ); if H0<1, update the value of H0 to (O j-1 +b j )÷U j-1 ; Go to S3; MAX{O j ,U j} means taking O j and U j The maximum value between the two;

[0024] S3, Update O j-1 The value of j , update U j-1 The value of U j ; If the value of variable j is less than N, increase the value of variable j by 1 and go to S2; if the value of variable j is equal to or greater than N, go to S4;

[0025] S4, record the value of H0 as the port bias value.

[0026] The beneficial effect of this step is that a large data storage system connected to a large number of storage devices has security risks such as malicious infection and network attacks during operation. Due to the closed nature of the storage system, these malicious activities are often implanted in the storage devices that have established a connection with the storage system, and enter the storage system at the same time with the transmission of data, causing abnormalities in both the storage device and the storage system. The data transmission speed and efficiency can best reflect whether the storage system or storage device has abnormalities. When the link is congested, the I / O performance is degraded, or traffic abnormalities occur, these problems directly affect the change in port bandwidth utilization. Therefore, the method of this step calculates the port deviation value through the activity data of the port. The port deviation value is calculated in order to use the value to represent the status benchmark of all port activities. The benchmark value is used to subsequently determine whether there is abnormal activity on the port. It can effectively detect and identify abnormal activities of each port in the storage system, timely discover and locate potential security risks, and take appropriate countermeasures before the risks spread, thereby fully ensuring the overall security and stability of the storage system.

[0027] The principle of this step is: for multiple ports, to obtain accurate status reference values ​​(i.e., port bias values), it is necessary to traverse the activity data of all ports. This step starts from the array Bur(P) corresponding to the port with the largest total bandwidth utilization through left and right iteration. The initial port bias value is set to the ratio of the sum of all values ​​greater than the average value in the array to the sum of all values ​​less than the average value in the array (O P ÷U P ), this is because the ratio can reflect the balance of the data at both ends of the mean, and at the same time update the balance amount by traversing all arrays, and continuously update the port bias value in combination with the balance amount to make it gradually accurate; it can be seen that the balance amount b j Constantly meet b j >0, and the value of H0 in each round is always close to 1 but never reaches 1 (except at the beginning P =U PIn addition, this situation only exists in theory and is rarely encountered in practice. When H0 of the previous round is greater than 1, the balancing amount will act on the denominator to further reduce H0 (on the right side of 1, so it needs to be reduced and close to 1). When H0 is less than 1, the balancing amount will act on the numerator to further increase it (on the left side of 1, so it needs to be increased and close to 1). The purpose of iterating the port bias value to close to 1 is to take into account that the bandwidth utilization is a ratio with a value less than 1, and the benchmark close to 1 can be better combined with the port activity data to improve the accuracy of locating abnormal ports, so that it can provide higher-precision analysis performance for subtle changes in the activities of each port.

[0028] Furthermore, the method of locating the abnormal port by the port deviation value is as follows: Dpiv_Bur(i) represents the deviation interval of the array Bur(i), Dpiv_Bur(i)=[Mbur(i)-H0,Mbur(i)+H0] (closed interval);

[0029] All values ​​in the array Bur(i) that fall outside the deviation interval Dpiv_Bur(i) are accumulated and recorded as Db(i), and SUM(Dpiv_Bur(i)) represents the accumulated value of all values ​​in Bur(i) that fall within the deviation interval Dpiv_Bur(i);

[0030] The skew extreme ske(i) corresponding to Bur(i) is calculated as follows:

[0031] When Db(i) is not 0, the sum of the value obtained by dividing Db(i) by SUM(Dpiv_Bur(i)) and SUM(Dpiv_Bur(i)) is taken as the extreme value ske(i); when Db(i) is 0, ske(i) is also recorded as 0;

[0032] The extreme skew value ske(i) is used as the attribute of port Port(i), and the K ports with the highest extreme skew values ​​are marked as ports with abnormal activities.

[0033] Furthermore, the value of K is set to K=[N÷H0]+1, where [N÷H0] represents rounding the value obtained by N÷H0.

[0034] The beneficial effect of this step is: by determining the deviation interval with the port deviation value and taking the ratio of the values ​​outside and inside the deviation interval as a measure of the abnormality of the port activity, it is possible to effectively identify ports with abnormal activity data, and provide a reliable early warning basis for the security monitoring of the storage system.

[0035] The principle of this step is: after determining the deviation interval, the value in the deviation interval reflects the current transmission status of the port. When the bandwidth utilization data in the deviation interval (SUM(Dpiv_Bur(i))) and the bandwidth utilization data outside the deviation interval (Db(i)) are in a proportional state (whether it is high load or low load, the ratio will never be unbalanced), and when Db(i) is very small and SUM(Dpiv_Bur(i)) is very large (indicating that the transmission status of the port is unbalanced between high and low loads), the value of ske(i) will be large, and when Db(i) is large and SUM(Dpiv_Bur(i)) is small (also indicating imbalance), the value of ske(i) will still be large. will be larger, therefore, the deviation value ske(i) from high to low can be used as the basis for sorting the degree of port abnormality. When ske(i) is 0, it means that the data in Bur(i) are relatively close. At this time, the deviation interval will include all the data in the array, and Db(i)=0 will appear accordingly (that is, no value falls outside the interval). At this time, if ske(i) is still calculated according to the original formula, a higher value will be presented, which will be mistakenly identified as an abnormal port. The data in Bur(i) are relatively close (regardless of high load or low load), indicating that the port is in a stable transmission state. Therefore, setting ske(i) to 0 avoids misjudgment, thereby more accurately identifying and responding to the correct abnormal port.

[0036] Furthermore, the method for security monitoring of abnormal ports and security verification of the data sent by them is specifically: limiting the access rights of abnormal ports to the storage system, network isolation of the storage devices connected to the abnormal ports, and sandbox isolation and integrity verification of the data received from the abnormal ports in the storage system.

[0037] The beneficial effects of the present invention are as follows: the method can effectively improve the security and stability of the storage system, by acquiring and analyzing the data of each port of the storage system in real time, using the port deviation value to accurately locate the ports with potential anomalies or where anomalies have occurred, and ensuring the security of data transmission through timely monitoring and inspection, thereby fully reducing the data risks caused by potential security risks, improving the data storage system's ability to handle malicious attacks, and effectively responding to various potential threats in a big data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Shown is a flow chart of a big data security storage method based on abnormal warning. DETAILED DESCRIPTION

[0039] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0040] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0041] like Figure 1 The flowchart of a method for safe storage of big data based on abnormal warning according to the present invention is shown below. Figure 1 To illustrate a big data security storage method based on abnormal warning according to an implementation mode of the present invention.

[0042] The present invention proposes a big data security storage method based on abnormal early warning, the method comprising the following steps:

[0043] Monitor each port of the storage system and obtain the activity data of the port;

[0044] Calculate the port deviation value based on the port activity data, and locate the abnormal port through the port deviation value;

[0045] Perform security monitoring on abnormal ports and perform security verification on the data they send.

[0046] Furthermore, each port of the storage system refers to a data transmission port for data communication between the storage device and the storage system. After the storage device establishes a connection with the storage system through the data transmission port, data is sent from the storage device to the storage system to complete the data transmission; a single or multiple storage devices can establish a connection with the storage system through the same port.

[0047] Furthermore, the storage system is one of CubeFS, HDFS, and Ceph.

[0048] Furthermore, the method of monitoring each port of the storage system and obtaining the activity data of the port is as follows: let the total number of ports of the storage system be N, let Port(i) be the i-th port among the N ports, i is the sequence number, i=1,2,…,N;

[0049] Set a monitoring period, record the bandwidth utilization of the port every T minutes during the monitoring period, and obtain the bandwidth utilization of each port at multiple times (one time is generated every T minutes, thus forming multiple times), and use array Bur(i) to store the bandwidth utilization of Port(i) at multiple times;

[0050] Then the N ports correspond to N arrays Bur(1), Bur(2),…, Bur(N), and these N arrays are used as active data.

[0051] Furthermore, the bandwidth utilization of a port refers to the instantaneous bandwidth utilization of the port at any moment, or refers to the average bandwidth utilization of the port in a time period formed between any moment and the moment before the moment (the length of the formed time period is the above-mentioned T minutes), and the average bandwidth utilization in the time period is defined as the average value of the instantaneous bandwidth utilization at all moments per second in the time period.

[0052] Specifically, the monitoring period is set to 24 hours and T is set to 5 minutes, which corresponds to 289 moments (24×60÷5+1=289). The instantaneous bandwidth utilization of port Port(i) at these 289 moments is obtained, and the length of the array Bur(i) is 289 (the length is the number of values ​​contained in the array), i=1,2,…,N, that is, the active data Bur(1), Bur(2),…,Bur(N) consists of 289×N values.

[0053] Specifically, Zabbix is ​​used to obtain the instantaneous bandwidth utilization of the port at any time.

[0054] Furthermore, the method for calculating the port bias value according to the activity data of the port is specifically as follows:

[0055] S1, for the i-th array Bur(i) in the N arrays Bur(1), Bur(2), ..., Bur(N) of activity data, MBur(i) represents the average value of all values ​​in Bur(i), and O i is the sum of all values ​​in the array Bur(i) that are greater than MBur(i), denoted by U i is the sum of all values ​​in the array Bur(i) that are less than MBur(i);

[0056] Then the array Bur(1) corresponds to O1 and U1, the array Bur(2) corresponds to O2 and U2, ..., the array Bur(N) corresponds to O N and U N ;

[0057] In the N arrays Bur(1), Bur(2), ..., Bur(N), select the array with the largest sum of all values ​​in the array and record it as Bur(P), where P is one of the numbers 1, 2, ..., N. Bur(P) naturally corresponds to O P and U P ;

[0058] Set variable H0, and set the initial value of H0 to 0 P ÷U P , set variable j = 1, and use Bur(j) to represent the jth array corresponding to the value of variable j in N arrays Bur(1), Bur(2), …, Bur(N);

[0059] Definition j-1 =O P , U j-1 =U P (At this time O j-1 =O0=O P , U j-1 =U0=U P The purpose of this setting is to facilitate the subsequent update of the value of H0, because the value of H0 is only needed for the first time in the iteration. P and U P In the second and subsequent iterations, we need to rely on the O corresponding to Bur(j-1) in the previous round. j-1 and U j-1 ), go to S2;

[0060] S2, Bur(j) corresponds to O j and U j , record the balance quantity b j =|O j -U j |÷MAX{O j ,U j}, if H0≥1, then update the value of H0 to O j-1 ÷(U j-1 +b j ); if H0<1, update the value of H0 to (O j-1 +b j )÷U j-1 ; Go to S3; MAX{O j ,U j} means taking O j and U j The maximum value between the two;

[0061] S3, Update O j-1 The value of j , update U j-1 The value of U j; If the value of variable j is less than N, increase the value of variable j by 1 and go to S2; if the value of variable j is equal to or greater than N, go to S4;

[0062] S4, record the value of H0 as the port bias value.

[0063] Furthermore, the method of locating the abnormal port by the port deviation value is as follows: Dpiv_Bur(i) represents the deviation interval of the array Bur(i), Dpiv_Bur(i)=[Mbur(i)-H0,Mbur(i)+H0] (closed interval);

[0064] All values ​​in the array Bur(i) that fall outside the deviation interval Dpiv_Bur(i) are accumulated and recorded as Db(i), and SUM(Dpiv_Bur(i)) represents the accumulated value of all values ​​in Bur(i) that fall within the deviation interval Dpiv_Bur(i);

[0065] The skew extreme ske(i) corresponding to Bur(i) is calculated as follows:

[0066] When Db(i) is not 0, the sum of the value obtained by dividing Db(i) by SUM(Dpiv_Bur(i)) and SUM(Dpiv_Bur(i)) is taken as the extreme value ske(i); when Db(i) is 0, ske(i) is also recorded as 0;

[0067] The specific mathematical expression is as follows:

[0068] ;

[0069] Taking the extreme deviation ske(i) as the attribute of port Port(i), the N ports Port(1), Port(2), …, Port(N) correspond to N extreme deviations ske(1), ske(2), …, ske(N), and the K ports with the highest extreme deviations are marked as ports with abnormal activities.

[0070] Furthermore, the value of K is set to K=[N÷H0]+1, where [N÷H0] represents rounding the value obtained by N÷H0.

[0071] Furthermore, the method for security monitoring of abnormal ports and security verification of the data sent by them is specifically: limiting the access rights of abnormal ports to the storage system, network isolation of the storage devices connected to the abnormal ports, and sandbox isolation and integrity verification of the data received from the abnormal ports in the storage system.

[0072] Specifically, the abnormal port is monitored for security and the data sent by the abnormal port is checked for security, and also includes generating alarm information of the abnormal port and sending it to the communication device of the storage system administrator.

[0073] The present invention provides a big data security storage method based on abnormal warning, which obtains the activity data of the port by monitoring each port of the storage system, calculates the port deviation value according to the activity data of the port, locates the abnormal port by the port deviation value, performs security monitoring on the abnormal port and performs security verification on the data sent by it. The method can effectively improve the security and stability of the storage system, obtain and analyze the data of each port of the storage system in real time, accurately locate the port with potential abnormality or abnormality by using the port deviation value, and ensure the transmission security of data in a timely monitoring and inspection manner, fully reduce the data risk caused by potential safety hazards, improve the processing ability of the data storage system against malicious attacks, and effectively deal with various potential threats in the big data environment. Although the description of the present invention has been quite detailed and the embodiments have been described in particular, it is not intended to be limited to any of these details or embodiments or any special embodiments, so as to effectively cover the predetermined scope of the present invention. In addition, the above describes the present invention with the foreseeable embodiments of the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen can still represent equivalent changes of the present invention.

Claims

1. A big data security storage method based on abnormal warning, characterized in that: The method comprises the following steps: Monitor each port of the storage system and obtain the activity data of the port; Calculate the port deviation value based on the port activity data, and locate the abnormal port through the port deviation value; Conduct security monitoring on abnormal ports and perform security verification on the data sent from them; The method of monitoring each port of the storage system and obtaining the activity data of the port is as follows: let the total number of ports of the storage system be N, and let Port(i) be the i-th port among the N ports; set a monitoring period, and record the bandwidth utilization of the port every T minutes during the monitoring period, so as to obtain the bandwidth utilization of each port at multiple times, and use the array Bur(i) to store the bandwidth utilization of Port(i) at multiple times; then the N ports correspond to N arrays Bur(1), Bur(2), ..., Bur(N), and these N arrays are used as activity data; The method for calculating the port bias value based on the port activity data is as follows: S1, for the i-th array Bur(i) in the N arrays Bur(1), Bur(2), ..., Bur(N) of activity data, MBur(i) represents the average value of all values ​​in Bur(i), and O i is the sum of all values ​​in the array Bur(i) that are greater than MBur(i), denoted by U i is the sum of all values ​​in the array Bur(i) that are less than MBur(i); select the array with the largest sum of all values ​​in the array from the N arrays Bur(1), Bur(2), …, Bur(N) and record it as Bur(P); Set variable H0, and set the initial value of H0 to 0 P ÷U P , set variable j = 1, and use Bur(j) to represent the jth array corresponding to the value of variable j in N arrays Bur(1), Bur(2), ..., Bur(N); define O j-1 =O P , U j-1 =U P , go to S2; S2, balance amount b j =|O j -U j |÷MAX{O j ,U j }, if H0≥1, then update the value of H0 to O j-1 ÷(U j-1 +b j ); if H0<1, update the value of H0 to (O j-1 +b j )÷U j-1 ; Go to S3; MAX{O j ,U j } means taking O j and U j The maximum value between the two; S3, Update O j-1 The value of j , update U j-1 The value of U j ; If the value of variable j is less than N, increase the value of variable j by 1 and go to S2; if the value of variable j is equal to or greater than N, go to S4; S4, record the value of H0 as the port bias value.

2. According to the method of claim 1, the method is characterized in that: The ports of the storage system refer to the data transmission ports for data communication between the storage device and the storage system. After the storage device establishes a connection with the storage system through the data transmission port, data is sent from the storage device to the storage system to complete the data transmission. A single or multiple storage devices can establish a connection with the storage system through the same port.

3. According to the method of claim 1, the method is characterized in that: The bandwidth utilization of a port refers to the instantaneous bandwidth utilization of the port at any moment, or refers to the average bandwidth utilization of the port in a period between any moment and the moment before that moment. The average bandwidth utilization in the period is defined as the average value of the instantaneous bandwidth utilization at all moments per second in the period.

4. According to the method of claim 1, the big data security storage method based on abnormal early warning is characterized in that: The method of locating abnormal ports by port deviation is as follows: Dpiv_Bur(i) represents the deviation interval of array Bur(i), Dpiv_Bur(i)=[Mbur(i)-H0,Mbur(i)+H0]; All values ​​in the array Bur(i) that fall outside the deviation interval Dpiv_Bur(i) are accumulated and recorded as Db(i), and SUM(Dpiv_Bur(i)) represents the accumulated value of all values ​​in Bur(i) that fall within the deviation interval Dpiv_Bur(i); The bias extreme ske(i) corresponding to Bur(i) is calculated as follows: when Db(i) is not 0, the sum of the value obtained by dividing Db(i) by SUM(Dpiv_Bur(i)) and SUM(Dpiv_Bur(i)) is taken as the bias extreme ske(i); when Db(i) is 0, ske(i) is also recorded as 0; The extreme deviation ske(i) is used as the attribute of port Port(i), and the K ports with the highest extreme deviation are marked as ports with abnormal activities, where the value of K is set to K=[N÷H0]+1, and [N÷H0] means rounding the value obtained by N÷H0.

5. According to the method of claim 1, the big data security storage method based on abnormal early warning is characterized in that: The specific method for security monitoring of abnormal ports and security verification of the data they send is as follows: restricting the access rights of abnormal ports to the storage system, network isolation of the storage devices connected to the abnormal ports, and sandbox isolation and integrity verification of the data received from the abnormal ports in the storage system.

Images