Data security processing method based on distributed storage

Abstract

The present invention discloses a data security processing method based on distributed storage, including establishing a distributed storage architecture; counting the number of accesses to each audit data item within a set period and the search conditions when each audit data item is accessed, and using the fields corresponding to the search conditions with the largest number of occurrences as candidate fields; sorting each type of candidate fields, and calculating the degree of discreteness of the candidate fields; sorting each type of candidate fields by the degree of discreteness, selecting the median candidate field as the sharding key, sharding the audit data to obtain sharding data, and storing the sharding data in different distributed nodes. The present invention solves the problem of local overheating that is prone to occur in conventional data sharding, and ensures the security, reliability and durability of audit data in a distributed storage environment. It effectively addresses the data security and long-term storage issues in the audit process, and provides a strong technical guarantee for the audit work of enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of data security computing, and in particular to a data security processing method based on distributed storage. Background Art

[0002] Online auditing involves multiple systems, so it is not suitable for centralized databases, but more suitable for distributed storage. Data sharding is a way of distributed storage, which divides data into different shards according to sharding keys for storage. The bottleneck of distributed storage lies in data transmission, not storage space. Therefore, the choice of sharding key directly affects the sharding result. If the choice is inappropriate, some nodes may be frequently accessed, while some nodes are almost never accessed, causing some nodes to be under huge access pressure and become the performance bottleneck of the system.

[0003] In traditional data sharding, the selection of sharding keys is usually based on some complex algorithms (such as hash algorithms) or determined empirically, resulting in an imbalance in storage space or access volume among shards. To this end, the existing technology has adopted a method of using neural network models to select sharding keys, which essentially uses powerful computing power to simulate the storage space and access volume of sharding results to determine the appropriate sharding key. However, the original intention of a distributed system is to disperse storage space and computing power, so that a single individual does not need powerful computing power, so this method is not suitable for use in a distributed system.

[0004] To this end, it is urgent to find a way to determine the appropriate sharding key with low computing power requirements, so that the access volume of the sharding results can be as balanced as possible and prevent some nodes from becoming system performance bottlenecks. Summary of the invention

[0005] In the distributed storage of audit data in the prior art, the selection of shard keys is relatively difficult. An inappropriate shard key may cause some nodes to be frequently accessed, while some nodes are hardly accessed, causing some nodes to be under huge access pressure, which becomes a performance bottleneck of the system. The present invention provides a data security processing method based on distributed storage, which changes the selection of fields corresponding to shard keys from traditional content-driven to access-driven, and then the degree of discreteness can be calculated in a quantitative manner, which can ensure that the access pressure of each node after sharding is as balanced as possible, and no longer requires powerful computing power and neural network models to select shard keys.

[0006] The following is the technical solution of the present invention.

[0007] The data security processing method based on distributed storage includes the following steps:

[0008] S1: Establish a distributed storage architecture based on audit nodes and distributed nodes;

[0009] S2: Count the number of times each audit data item is accessed within a set period and the search conditions when each audit data item is accessed, and select the fields corresponding to the search conditions that appear the most times as candidate fields;

[0010] S3: sort each type of candidate field based on its access times in different audit data, and calculate the discreteness of the candidate field according to each access times, and repeat S3 until each type of candidate field obtains its own discreteness;

[0011] S4: Sort each type of candidate field by discreteness, select the median candidate field as the sharding key, shard the audit data based on the field value of the sharding key to obtain sharded data, and store the sharded data in different distributed nodes. Each distributed node stores at least two types of sharded data.

[0012] S5: The audit node queries and accesses the distributed nodes based on the field value of the shard key and the search condition, and records the access frequency. If the access frequency of the distributed nodes is higher than the threshold within the set period, S6 is executed to migrate the data. Otherwise, S5 is executed after the next period.

[0013] S6: Mark the distributed node with the highest access frequency as the outgoing node, mark the distributed node with the lowest access frequency as the incoming node, migrate a type of shard data from the outgoing node to the incoming node, and return to S5.

[0014] In this solution, the fields corresponding to the search conditions are first used as candidate fields. The search conditions are directly related to the data being accessed, so the method is more scientific. At the same time, this solution cleverly sorts the number of accesses and quantifies the audit data, and then the degree of discreteness of the candidate fields can be judged according to the number of accesses. Compared with traditional solutions that determine the sharding key through some complex algorithms (such as hash algorithms) or experience, this solution selects the sharding key through the degree of discreteness of the candidate fields. The method is simple and has low computing power requirements.

[0015] In this solution, the median candidate field is selected as the sharding key because the median candidate field is often more evenly distributed in data, that is, the frequency of occurrence of each data in the data set is relatively balanced. The candidate field with a lower degree of discreteness as the sharding key often results in too much sharding data of a certain type, which cannot achieve a balance in storage space or access volume, while the candidate field with a higher degree of discreteness as the sharding key often results in too many types of sharding data, which cannot achieve a balance in storage space or access volume. In addition, selecting the median candidate field as the sharding key is more flexible when expanding. When adding or deleting nodes, it is easier to adjust the sharding strategy to maintain a balanced distribution of storage space or access volume.

[0016] In this solution, the shard data is stored in different distributed nodes, and each distributed node stores at least two types of shard data. This design can effectively distribute query and access loads between nodes. Query requests can be processed in parallel on multiple nodes instead of being concentrated on a few nodes, avoiding the problem of single-point overload. At the same time, since each node stores multiple types of shard data, query requests can complete the aggregation of multiple types of data on a single node, reducing the need for cross-node communication and data transmission, thereby significantly reducing query latency and improving the system's response speed.

[0017] In this solution, since the shard data is stored in different distributed nodes and each distributed node stores at least two types of shard data, the shard data can be easily migrated. If the access frequency of a distributed node is higher than the threshold within a set period, the shard data migration step can be executed. Compared with the prior art, when a node accesses the upper limit of the frequency, it is necessary to reorganize the data of the entire node on a large scale. Only part of the data needs to be migrated to achieve the purpose of reducing the access frequency of a distributed node. It can also increase the access frequency of distributed nodes with lower access frequency, so that the access frequency between distributed nodes can be further balanced, thereby improving the efficiency of the overall distributed storage.

[0018] Preferably, the S3 is specifically:

[0019] For each type of candidate field, calculate the number of accesses in different audit data using the candidate field as the search condition, sort them according to the value of each access number, and calculate the degree of dispersion of this type of candidate field through variance, standard deviation or interquartile range, and repeat S3 until each type of candidate field obtains its own degree of dispersion.

[0020] In this solution, the variance is the average of the squares of the differences between each data and its mean, and the standard deviation is the square root of the variance. The principle and result of calculating the degree of dispersion of the candidate field by variance or standard deviation are the same; the interquartile range is the difference between the third quartile (Q3) and the first quartile (Q1), which is used to measure the degree of dispersion of the intermediate data. Variance and standard deviation are indicators of the overall dispersion of data, which are sensitive to extreme values. The interquartile range is an indicator of the dispersion of the middle part of the data, which is not sensitive to extreme values ​​and can be flexibly selected and adjusted in actual application.

[0021] Preferably, in S4, the audit data is fragmented based on the field value of the fragmentation key to obtain fragmented data specifically as follows:

[0022] Get all different field values ​​of the shard key, find the audit data that matches each field value, and classify all audit data that matches the field value into the same type of shard data. The number of field values ​​is consistent with the number of types of shard data.

[0023] Preferably, the S3 is specifically:

[0024] For each type of candidate field, calculate the number of accesses in different audit data using the candidate field as the search condition, sort them according to the value of each access number, and calculate the degree of dispersion of this type of candidate field through variance and interquartile range. Repeat S3 until each type of candidate field obtains its own degree of dispersion, which includes two degrees of dispersion calculated by variance and interquartile range.

[0025] The S4 is specifically:

[0026] Sort each type of candidate fields by degree of discreteness, select the median candidate field as the sharding key, and determine whether the candidate fields selected by the two degrees of discreteness are the same. If they are the same, shard the audit data based on the field value of the sharding key to obtain sharded data, and store the sharded data in different distributed nodes. Each distributed node stores at least two types of sharded data. If they are not the same, select the candidate field corresponding to one of the degrees of discreteness as the sharding key.

[0027] Preferably, if they are not the same, then one of the candidate fields corresponding to the discrete degree is selected as the sharding key, and the principle of selecting one of the discrete degrees is to select one of the candidate fields corresponding to the discrete degree as the sharding key according to the principle of the smoothest access frequency.

[0028] In this scheme, the method of choosing variance, standard deviation or interquartile range is considered to be different from the manual method. By calculating and judging which method corresponds to the smoothest access frequency, the problem of manual selection being insensitive to data and the inaccurate calculation method of the selected discrete degree can be avoided.

[0029] Preferably, the candidate fields corresponding to one of the discrete degrees are selected as the sharding key according to the principle of the smoothest access frequency, specifically:

[0030] The audit data is sampled and sharded, and the sharded data is stored; the sharded data corresponding to each discrete degree is tested, and the audit node queries and accesses the distributed nodes based on the field value and search conditions of the sharding key, and records the access frequency, selects the discrete degree corresponding to the smoothest access frequency, and uses the alternative field corresponding to the discrete degree as the sharding key.

[0031] Preferably, the S6 is replaced by marking the distributed node with the highest access frequency as the outgoing node, marking the distributed node with the lowest access frequency as the incoming node, migrating a type of shard data of the outgoing node to the incoming node, and after the migration is completed, judging whether the access frequency of the incoming node is higher than the threshold, if it is higher than the threshold, canceling the migration process, and trying to migrate other types of shard data of the outgoing node to the incoming node, until it is judged that the access frequency of the incoming node is lower than the threshold, and then returning to S5.

[0032] Preferably, if the access frequency of the incoming node is still higher than the threshold after all categories of shard data of the outgoing node are attempted to be migrated to the incoming node, the incoming node is adjusted, and a category of shard data of the incoming node is migrated to other distributed nodes with lower access frequencies, and then step S6 is executed to make the access frequency of the incoming node lower than the threshold.

[0033] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of the above-mentioned data security processing method based on distributed storage are implemented.

[0034] The present invention also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the above-mentioned data security processing method based on distributed storage are implemented.

[0035] The substantial effects of the present invention include: the present invention solves the local overheating problem that is easy to occur in conventional data sharding, by changing the selection of the field corresponding to the sharding key from the traditional content-driven to the access-number-driven, and then the discrete degree can be calculated in a quantitative manner, which can ensure that the access pressure of each node after sharding is as balanced as possible, and scientific and reasonable sharding keys are used to shard the audit data to ensure the security, reliability and durability of the audit data in a distributed storage environment. The solution of the present invention effectively addresses the data security and long-term storage issues in the audit process, and provides a strong technical guarantee for the audit work of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution will be clearly and completely described below in combination with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0039] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0040] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0041] The technical solution of the present invention is described in detail with specific embodiments below. The embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0042] Embodiment 1: Data security processing method based on distributed storage, such as Figure 1 As shown, the following steps are included:

[0043] S1: Establish a distributed storage architecture based on audit nodes and distributed nodes;

[0044] S2: Count the number of times each audit data item is accessed within a set period and the search conditions when each audit data item is accessed, and select the fields corresponding to the search conditions that appear the most times as candidate fields;

[0045] S3: sort each type of candidate field based on its access times in different audit data, and calculate the discreteness of the candidate field according to each access times, and repeat S3 until each type of candidate field obtains its own discreteness;

[0046] S4: Sort each type of candidate field by discreteness, select the median candidate field as the sharding key, shard the audit data based on the field value of the sharding key to obtain sharded data, and store the sharded data in different distributed nodes. Each distributed node stores at least two types of sharded data.

[0047] S5: The audit node queries and accesses the distributed nodes based on the field value of the shard key and the search condition, and records the access frequency. If the access frequency of the distributed nodes is higher than the threshold within the set period, S6 is executed to migrate the data. Otherwise, S5 is executed after the next period.

[0048] S6: Mark the distributed node with the highest access frequency as the outgoing node, mark the distributed node with the lowest access frequency as the incoming node, migrate a type of shard data from the outgoing node to the incoming node, and return to S5.

[0049] In this solution, the fields corresponding to the search conditions are first used as candidate fields. The search conditions are directly related to the data being accessed, so the method is more scientific. At the same time, this solution cleverly sorts the number of accesses and quantifies the audit data, and then the degree of discreteness of the candidate fields can be judged based on the number of accesses. Compared with the traditional solution that uses some complex algorithms (such as hash algorithms) or determines the sharding key empirically, this solution changes the selection of the field corresponding to the sharding key from the traditional content-driven to the access-driven, and then the degree of discreteness can be calculated in a quantitative way, which can ensure that the access pressure of each node after sharding is as balanced as possible. The method is simple and has low computing power requirements.

[0050] In this solution, the median candidate field is selected as the sharding key because the median candidate field is often more evenly distributed in data, that is, the frequency of occurrence of each data in the data set is relatively balanced. The candidate field with a lower degree of discreteness as the sharding key often results in too much sharding data of a certain type, which cannot achieve a balance in storage space or access volume, while the candidate field with a higher degree of discreteness as the sharding key often results in too many types of sharding data, which cannot achieve a balance in storage space or access volume. In addition, selecting the median candidate field as the sharding key is more flexible when expanding. When adding or deleting nodes, it is easier to adjust the sharding strategy to maintain a balanced distribution of storage space or access volume.

[0051] In this solution, the shard data is stored in different distributed nodes, and each distributed node stores at least two types of shard data. This design can effectively distribute query and access loads between nodes. Query requests can be processed in parallel on multiple nodes instead of being concentrated on a few nodes, avoiding the problem of single-point overload. At the same time, since each node stores multiple types of shard data, query requests can complete the aggregation of multiple types of data on a single node, reducing the need for cross-node communication and data transmission, thereby significantly reducing query latency and improving the system's response speed.

[0052] In this solution, since the shard data is stored in different distributed nodes and each distributed node stores at least two types of shard data, the shard data can be easily migrated. If the access frequency of a distributed node is higher than the threshold within a set period, the shard data migration step can be executed. Compared with the prior art, when a node accesses the upper limit of the frequency, it is necessary to reorganize the data of the entire node on a large scale. Only part of the data needs to be migrated to achieve the purpose of reducing the access frequency of a distributed node. It can also increase the access frequency of distributed nodes with lower access frequency, so that the access frequency between distributed nodes can be further balanced, thereby improving the efficiency of the overall distributed storage.

[0053] The S3 is specifically:

[0054] For each type of candidate field, calculate the number of accesses in different audit data using the candidate field as the search condition, sort them according to the value of each access number, and calculate the degree of dispersion of the candidate field of this type through variance, standard deviation or interquartile difference, and repeat S3 until each type of candidate field obtains its own degree of dispersion. In this scheme, variance is the average of the squares of the differences between each data and its mean, and standard deviation is the square root of variance. The principle and result of calculating the degree of dispersion of the candidate field by variance or standard deviation are the same; interquartile difference is the difference between the third quartile (Q3) and the first quartile (Q1), which is used to measure the degree of dispersion of the intermediate data. Variance and standard deviation are indicators for measuring the overall degree of dispersion of data, which are sensitive to extreme values. Interquartile difference is an indicator for measuring the degree of dispersion of the middle part of the data, which is not sensitive to extreme values. It can be flexibly selected and adjusted in actual application.

[0055] In S4, the audit data is fragmented based on the field value of the fragmentation key to obtain the fragmented data specifically as follows:

[0056] Get all different field values ​​of the shard key, find the audit data that matches each field value, and classify all audit data that matches the field value into the same type of shard data. The number of field values ​​is consistent with the number of types of shard data.

[0057] Specifically, the present invention is as follows in practical application:

[0058] There are 1,000 audit data items in total, and the audit data is stored in a table. Each row in the table represents an audit data item, and each column represents a subdivision of the audit data, including name, number, generation date, creator, modification date, modifier, category, etc. Table 1 is an example table of audit data, showing a specific row of audit data categories:

[0059] Table 1

[0060]

[0061] The number of times each audit data item is accessed within the set period and the search conditions when each audit data item is accessed are counted. This solution selects the fields corresponding to the top five search conditions as candidate fields. In actual statistics, the number of times the audit data with name as the search condition is accessed is 1720 times, the number of times the audit data with number as the search condition is accessed is 4138 times, the number of times the audit data with generation date as the search condition is accessed is 5438 times, the number of times the audit data with creator as the search condition is accessed is 3665 times, the number of times the audit data with modification date as the search condition is accessed is 4285 times, the number of times the audit data with modifier as the search condition is accessed is 2814 times, and the number of times the audit data with category as the search condition is accessed is 6727 times. Therefore, the top five search conditions correspond to category, generation date, modification date, number and creator, and the fields corresponding to these search conditions are used as candidate fields.

[0062] For the category candidate field, we sorted them based on the number of accesses in different audit data. The total number of accesses was 6727. Then, we calculated the degree of dispersion of the candidate field according to the number of accesses for each item. The degree of dispersion was determined by the variance, and the calculated variance was 0.045. The calculation method for other candidate fields was the same as that for the category candidate field. The calculated variances were: creation date 0.033, modification date 0.082, number 0.016, creator 0.023.

[0063] Sort each type of candidate field by discreteness, select the median candidate field as the sharding key. In this embodiment, the median is the generation date, so the generation date is used as the sharding key, and the audit data is sharded based on the field value of the generation date to obtain sharded data. It should be noted that since the data of the generation date includes year, month, day, hour, and minute, overly detailed classification will lead to too many types of sharded data, affecting the efficiency of sharding. Therefore, the same year, month, and day are selected for sharding to obtain sharded data, and the year, month, and day of the generation date in each type of sharded data are the same. The principle and method of using the modification date as the sharding key are roughly the same as the generation date as the sharding key, and for example, the number, creator, and modifier will be sharded according to different actual values ​​or different people to obtain sharded data, and the number, creator, or modifier in each type of sharded data is the same.

[0064] Embodiment 2: A data security processing method based on distributed storage, the principle and implementation method of which are substantially the same as those of Embodiment 1, except that S3 is specifically:

[0065] For each type of candidate field, calculate the number of accesses in different audit data using the candidate field as the search condition, sort them according to the value of each access number, and calculate the degree of dispersion of this type of candidate field through variance and interquartile range, repeat S3 until each type of candidate field obtains its own degree of dispersion, and the degree of dispersion includes two degrees of dispersion calculated by variance and interquartile range; because the variance and standard deviation have the same results in judging the degree of dispersion in actual applications, only variance and interquartile range are selected for comparison in this scheme.

[0066] The S4 is specifically:

[0067] Sort each type of candidate fields by degree of discreteness, select the median candidate field as the sharding key, and determine whether the candidate fields selected by the two degrees of discreteness are the same. If they are the same, shard the audit data based on the field value of the sharding key to obtain sharded data, and store the sharded data in different distributed nodes. Each distributed node stores at least two types of sharded data. If they are not the same, select the candidate field corresponding to one of the degrees of discreteness as the sharding key.

[0068] If they are not the same, then one of the candidate fields corresponding to the discrete degree is selected as the sharding key. The principle of selecting one of the discrete degrees is to select one of the candidate fields corresponding to the discrete degree as the sharding key based on the principle of the smoothest access frequency.

[0069] In this scheme, the method of choosing variance, standard deviation or interquartile range is considered to be different from the manual method. By calculating and judging which method corresponds to the smoothest access frequency, the problem of manual selection being insensitive to data and the inaccurate calculation method of the selected discrete degree can be avoided.

[0070] In this solution, the method for judging the degree of discreteness is not limited to variance, standard deviation or interquartile range. The degree of discreteness can also be judged by range or coefficient of dispersion. If they are different, one of the candidate fields corresponding to the degree of discreteness is selected as the sharding key. The median with the consistent results of most methods for judging the degree of discreteness can be selected as the sharding key. If the results of all methods for judging the degree of discreteness are inconsistent, one of the candidate fields corresponding to the degree of discreteness is selected as the sharding key based on the principle of the smoothest access frequency.

[0071] The above-mentioned principle of selecting one of the candidate fields corresponding to the discrete degree as the sharding key according to the principle of the smoothest access frequency is as follows:

[0072] The audit data is sampled and sharded, and the sharded data is stored; the sharded data corresponding to each discrete degree is tested, and the audit node queries and accesses the distributed nodes based on the field value and search conditions of the sharding key, and records the access frequency, selects the discrete degree corresponding to the smoothest access frequency, and uses the alternative field corresponding to the discrete degree as the sharding key.

[0073] In this solution, by testing the shard data corresponding to each discrete degree, the discrete degree corresponding to the smoothest access frequency can be obtained. When testing the shard data corresponding to each discrete degree, it should be ensured that the number of tests is large and the test should cover as many search conditions as possible. If conditions require multiple tests, the optimal choice that meets most tests is determined during multiple tests, and the candidate field corresponding to the corresponding discrete degree is used as the shard key.

[0074] Example 3: A data security processing method based on distributed storage, whose principle and implementation method are basically the same as those of Example 1, except that S6 is replaced by marking the distributed node with the highest access frequency as the outgoing node, and marking the distributed node with the lowest access frequency as the incoming node, migrating a type of shard data of the outgoing node to the incoming node, and after the migration is completed, judging whether the access frequency of the incoming node is higher than the threshold, if it is higher than the threshold, canceling the migration process, and trying to migrate other types of shard data of the outgoing node to the incoming node, until it is judged that the access frequency of the incoming node is lower than the threshold, and then returning to S5.

[0075] If the access frequency of the incoming node is still higher than the threshold after all categories of shard data of the outgoing node are attempted to be migrated to the incoming node, the incoming node is adjusted, and a category of shard data of the incoming node is migrated to other distributed nodes with lower access frequencies, and then step S6 is executed to make the access frequency of the incoming node lower than the threshold.

[0076] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of the above-mentioned data security processing method based on distributed storage are implemented.

[0077] The present invention also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the above-mentioned data security processing method based on distributed storage are implemented.

[0078] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0079] In the embodiments provided in the present application, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the embodiments of the structure described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another structure, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0081] In addition, each functional unit in the embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0083] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data security processing method based on distributed storage, characterized in that: The following steps are involved: S1: Establish a distributed storage architecture based on audit nodes and distributed nodes; S2: Count the number of times each audit data item is accessed within a set period and the search conditions when each audit data item is accessed, and select the fields corresponding to the search conditions that appear the most times as candidate fields; S3: sort each type of candidate field based on its access times in different audit data, and calculate the discreteness of the candidate field according to each access times, and repeat S3 until each type of candidate field obtains its own discreteness; S4: Sort each type of candidate field by discreteness, select the median candidate field as the sharding key, shard the audit data based on the field value of the sharding key to obtain sharded data, and store the sharded data in different distributed nodes. Each distributed node stores at least two types of sharded data. S5: The audit node queries and accesses the distributed nodes based on the field value of the shard key and the search condition, and records the access frequency. If the access frequency of the distributed nodes is higher than the threshold within the set period, S6 is executed to migrate the data. Otherwise, S5 is executed after the next period. S6: Mark the distributed node with the highest access frequency as the outgoing node, mark the distributed node with the lowest access frequency as the incoming node, migrate a type of shard data from the outgoing node to the incoming node, and return to S5.

2. The data security processing method based on distributed storage according to claim 1 is characterized in that: The S3 is specifically: For each type of candidate field, calculate the number of accesses in different audit data using the candidate field as the search condition, sort them according to the value of each access number, and calculate the degree of dispersion of this type of candidate field through variance, standard deviation or interquartile range, and repeat S3 until each type of candidate field obtains its own degree of dispersion.

3. The data security processing method based on distributed storage according to claim 2 is characterized in that: In S4, the audit data is fragmented based on the field value of the fragmentation key to obtain the fragmented data specifically as follows: Get all different field values ​​of the shard key, find the audit data that matches each field value, and classify all audit data that matches the field value into the same type of shard data. The number of field values ​​is consistent with the number of types of shard data.

4. The data security processing method based on distributed storage according to claim 1 is characterized in that: The S3 is specifically: For each type of candidate field, calculate the number of accesses in different audit data using the candidate field as the search condition, sort the fields according to the value of each access number, and calculate the degree of dispersion of the candidate field using the variance and the interquartile range. Repeat S3 until each type of candidate field obtains its own degree of dispersion, including two degrees of dispersion calculated using the variance and the interquartile range. The S4 is specifically: Sort each type of candidate fields by degree of discreteness, select the median candidate field as the sharding key, and determine whether the candidate fields selected by the two degrees of discreteness are the same. If they are the same, shard the audit data based on the field value of the sharding key to obtain sharded data, and store the sharded data in different distributed nodes. Each distributed node stores at least two types of sharded data. If they are not the same, select the candidate field corresponding to one of the degrees of discreteness as the sharding key.

5. The data security processing method based on distributed storage according to claim 4 is characterized in that: If they are not the same, then one of the candidate fields corresponding to the discrete degree is selected as the sharding key. The principle of selecting one of the discrete degrees is to select one of the candidate fields corresponding to the discrete degree as the sharding key based on the principle of the smoothest access frequency.

6. The data security processing method based on distributed storage according to claim 5 is characterized in that: The above-mentioned principle of selecting one of the candidate fields corresponding to the discrete degree as the sharding key according to the principle of the smoothest access frequency is as follows: The audit data is sampled and sharded, and the sharded data is stored; the sharded data corresponding to each discrete degree is tested, and the audit node queries and accesses the distributed nodes based on the field value and search conditions of the sharding key, and records the access frequency, selects the discrete degree corresponding to the smoothest access frequency, and uses the alternative field corresponding to the discrete degree as the sharding key.

7. The data security processing method based on distributed storage according to claim 1 is characterized in that: The S6 is replaced by marking the distributed node with the highest access frequency as the outgoing node, marking the distributed node with the lowest access frequency as the incoming node, migrating a type of shard data of the outgoing node to the incoming node. After the migration is completed, it is determined whether the access frequency of the incoming node is higher than the threshold. If it is higher than the threshold, the migration process is canceled, and other types of shard data of the outgoing node are attempted to be migrated to the incoming node until it is determined that the access frequency of the incoming node is lower than the threshold, and then returning to S5.

8. The data security processing method based on distributed storage according to claim 1 is characterized in that: If the access frequency of the incoming node is still higher than the threshold after all categories of shard data of the outgoing node are attempted to be migrated to the incoming node, the incoming node is adjusted, and a category of shard data of the incoming node is migrated to other distributed nodes with lower access frequencies, and then step S6 is executed to make the access frequency of the incoming node lower than the threshold.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the data security processing method based on distributed storage as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the data security processing method based on distributed storage as described in any one of claims 1 to 8 are implemented.

Images