Automatic capacity management method and system for distributed data analysis engine

Abstract

The invention relates to the technical field of distributed data analysis engines, in particular to an automatic capacity management method and system for a distributed data analysis engine, and the method comprises the steps: S1, dividing to-be-stored different data into data groups with the number of characteristic numerical values; s2, respectively storing all data in the plurality of data groups into corresponding node modules in a secret storage mode; s3, judging whether the number of currently used node modules needs to be adjusted or not; s4, if the number of the currently used node modules is reduced, transferring all secret data in the data sets with the number of difference values between the first ratio and the second ratio to other corresponding node modules, and otherwise, transferring all secret data in the data sets with the number of difference values between the third ratio and the fourth ratio to corresponding node modules to be added; and S5, carrying out recovery processing on the secret data, and carrying out analysis processing by using the data subjected to the recovery processing. The capacity can be effectively utilized.

Description

Technical Field

[0001] The present application relates to the technical field of distributed data analysis engines, and in particular to a method and system for automatically managing the capacity of a distributed data analysis engine. Background Art

[0002] A distributed data analysis engine refers to a system that can process and analyze large amounts of data in a distributed storage environment. It is widely used in scenarios such as big data processing, real-time data analysis, and machine learning.

[0003] A Chinese patent application with publication number CN112099977A discloses a real-time data analysis engine for a distributed tracking system, including a data processing module and a data acquisition module and a data analysis module connected to the data processing module. The data acquisition module and the data processing module are connected through a data receiving module. A Kafka cluster is used as an intermediate layer between the data receiving module and the data processing module. An asynchronous transmission method is used between Kafka and the data receiving node in the data receiving module. Kafka counts the response time of each URL access within a specific time interval. The data processing module uses data pre-aggregation based on a time window, aggregates data by comparing fields corresponding to the response time, and adds new data to the aggregated result. The result of data pre-aggregation is stored in a Redis cache, the data summary result is extracted and stored, and the data in the Redis cache is deleted at the same time. In addition, the Chinese patent application with publication number CN111625696A provides a distributed scheduling method, computing node device, readable storage medium, computing equipment and distributed scheduling system of a multi-source data analysis engine, which saves a lot of communication costs and improves the distributed processing efficiency of the multi-source data analysis engine. The method includes: the computing node with the highest scheduling index of the multi-source data analysis engine receives the query task; the computing node determines the sub-query task of the query task including the intermediate result set, and determines the storage node of the intermediate result set; the computing node calculates the first time cost of migrating the intermediate result set to the local, and calculates the second time cost of executing the sub-query task by the storage node; the computing node selects whether the computing node migrates the intermediate result set to the local and executes the sub-query task, or the storage node executes the sub-query task according to the comparison result of the first time cost and the second time cost. However, the above two patent applications do not consider how to effectively use capacity, and the capacity management efficiency is not high. Summary of the invention

[0004] The management module of the present application divides different data to be stored into data groups with the same number of characteristic values, so that each node module of the present application can store secret data in a balanced manner, and when the number of node modules currently in use needs to be adjusted, the management module of the present application performs capacity management in units of data groups. The present application aims to effectively utilize capacity and improve capacity management efficiency.

[0005] The present application provides a method for automatically managing the capacity of a distributed data analysis engine, comprising the following steps:

[0006] S1. The management module obtains a number of preset values, determines characteristic values ​​based on the obtained preset values, and divides different data to be stored into data groups with the same number of characteristic values;

[0007] S2, the management module formulates a storage plan table for different data to be stored based on the number of node modules currently in use and the number of all data groups, the number of node modules currently in use corresponds to one of several preset values, and stores all data in several data groups in the corresponding node modules in a secret storage manner according to the storage plan table;

[0008] S3, the management module determines whether the number of currently used node modules needs to be adjusted, and if not, repeats this step, and if yes, proceeds to the next step;

[0009] S4. In the case of reducing the number of node modules currently in use, the management module calculates a first ratio of the characteristic value to the number of node modules after adjustment, calculates a second ratio of the characteristic value to the number of node modules before adjustment, and transfers all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules; in the case of increasing the number of node modules currently in use, the management module calculates a third ratio of the characteristic value to the number of node modules before adjustment, calculates a fourth ratio of the characteristic value to the number of node modules after adjustment, and transfers all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added;

[0010] S5. The extraction module extracts the stored secret data from the corresponding node module, performs recovery processing on the secret data, and the analysis module uses the recovered data for analysis processing.

[0011] As a preferred technical solution of the present application, in S4, when the number of currently used node modules is reduced, the reduced number of node modules corresponds to one of several preset values.

[0012] As a preferred technical solution of the present application, in S4, when the number of currently used node modules is increased, the increased number of node modules corresponds to one of several preset values.

[0013] As a preferred technical solution of the present application, the management module stores all data in several data groups in corresponding node modules in a secret storage manner, including the following steps:

[0014] S21, for each data in the plurality of data groups, the management module performs secret processing on the data to obtain secret data;

[0015] S22, with respect to the secret data to be stored corresponding to each data in the plurality of data groups, the management module compares the secret data to be stored with the different secret data stored in the corresponding node module;

[0016] S23. With respect to the secret data to be stored corresponding to each data in several data groups, the management module determines whether there is stored secret data that has the same part as the secret data to be stored. If so, storage processing is performed on different parts of the secret data to be stored; if not, storage processing is performed on the secret data to be stored.

[0017] As a preferred technical solution of the present application, in S21, the management module performs confidentiality processing on the data to obtain confidential data, including the following steps:

[0018] S211, the management module divides the data into a plurality of partial data, and the management module sequentially generates a sequence number for each partial data in the order in which the plurality of partial data are obtained;

[0019] S212, the management module uses all the partial data except the partial data corresponding to the largest sequence number among the plurality of partial data to form intermediate data, and performs a first operation processing on the intermediate data to obtain first operation result data;

[0020] S213, the management module performs a second operation on the first operation result data and the partial data corresponding to the largest sequence number among the plurality of partial data to obtain second operation result data, and generates obfuscated data according to the second operation result data;

[0021] S214, the management module performs a second operation on the obfuscated data and the intermediate data to obtain the first secret data, and performs encryption on the partial data corresponding to the largest sequence number among the partial data to obtain the second secret data, and connects the first secret data and the second secret data to obtain the secret data.

[0022] As a preferred technical solution of the present application, in S22, the management module compares the secret data to be stored with the different secret data stored in the corresponding node module, including the following steps:

[0023] S221, with respect to each secret data stored in the corresponding node module, the management module performs segmentation processing on the secret data to obtain a first portion of data and a second portion of data, and performs the same segmentation processing on the secret data to be stored to obtain a third portion of data and a fourth portion of data;

[0024] S222, the management module determines whether the first part of data is the same as the third part of data. If yes, the third part of data is regarded as the same part of the secret data to be stored, and the fourth part of data is regarded as the different part of the secret data to be stored.

[0025] As a preferred technical solution of the present application, the extraction module performs recovery processing on the secret data, including the following steps:

[0026] S51, the extraction module performs segmentation processing on the secret data to obtain partial secret data 1 and partial secret data 2, and continues to perform decryption processing on the partial secret data 2 to obtain decryption result data;

[0027] S52, the extraction module generates mixed data according to the decryption result data, and performs a second operation on the mixed data and part of the secret data to obtain first restored data;

[0028] S53, the extraction module performs a first operation on the first recovery data to obtain intermediate recovery data, and continues to perform a second operation on the intermediate recovery data and the decryption result data to obtain second recovery data, and connects the first recovery data and the second recovery data to obtain the recovered data.

[0029] As a preferred technical solution of the present application, the extraction module extracts metadata of the secret data while extracting the stored secret data from the corresponding node module. Extracting the metadata of the secret data includes: judging whether the amount of metadata in the memory exceeds a preset threshold value, and if not, extracting the metadata of the secret data; if exceeded, unloading from the memory several metadata that have not been accessed in the last 7 days, and continuing to judge whether the amount of metadata in the memory exceeds the preset threshold value, and if not exceeded, extracting the metadata of the secret data; if exceeded, determining several metadata using the LRU algorithm, unloading the determined several metadata from the memory, and extracting the metadata of the secret data.

[0030] The present application also provides a capacity automatic management system for a distributed data analysis engine, including the following modules:

[0031] A management module is used to obtain a number of preset values, determine characteristic values ​​based on the obtained preset values, and divide different data to be stored into data groups with the same number of characteristic values; at the same time, it is used to formulate a storage plan for different data to be stored based on the number of node modules currently in use and the number of all data groups, the number of node modules currently in use corresponds to one of the several preset values, and all data in the several data groups are stored in the corresponding node modules in a secret storage manner according to the storage plan; and it is used to determine whether the number of node modules currently in use needs to be adjusted; and it is also used to reduce the number of node modules currently in use. , calculating a first ratio of the characteristic value to the number of node modules after adjustment, calculating a second ratio of the characteristic value to the number of node modules before adjustment, transferring all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules, and in the case of increasing the number of node modules currently in use, calculating a third ratio of the characteristic value to the number of node modules before adjustment, calculating a fourth ratio of the characteristic value to the number of node modules after adjustment, transferring all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added;

[0032] A node module, used to store secret data corresponding to the data to be stored;

[0033] An extraction module, used to extract the stored secret data from the corresponding node module and perform recovery processing on the secret data;

[0034] The analysis module is used to perform analysis on the data recovered by the extraction module.

[0035] Compared with the prior art, the beneficial effects of the present application are at least as follows:

[0036] In the technical solution provided in the present application, first, a number of preset values ​​are obtained, and characteristic values ​​are determined based on the obtained preset values, and different data to be stored are divided into data groups with the same number of characteristic values; secondly, a storage plan table for different data to be stored is formulated based on the number of node modules currently in use and the number of all data groups, and all data in the several data groups are stored in corresponding node modules in a secret storage manner according to the storage plan table; thirdly, in the case where it is necessary to reduce the number of node modules currently in use, a first ratio of the characteristic value to the number of node modules after adjustment is calculated, and a second ratio of the characteristic value to the number of node modules before adjustment is calculated, and the ... Separately transfer all secret data in the data group of the difference number of the first ratio and the second ratio from the node module to be reduced to the corresponding other node modules. When it is necessary to increase the number of node modules currently in use, calculate the third ratio of the characteristic value to the number of node modules before adjustment, calculate the fourth ratio of the characteristic value to the number of node modules after adjustment, and transfer all secret data in the data group of the difference number of the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added; finally, extract the stored secret data from the corresponding node module, perform recovery processing on the extracted secret data, and use the recovered data for analysis and processing. Through this application, not only can the capacity be effectively utilized and the capacity management efficiency be improved, but also the problem of data leakage can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0038] Figure 1 A flow chart of a method for automatic capacity management of a distributed data analysis engine in this application;

[0039] Figure 2 A schematic diagram of an automatic capacity management system for a distributed data analysis engine in this application. DETAILED DESCRIPTION

[0040] Embodiments of the present application provide a method and system for automatic capacity management of a distributed data analysis engine. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In an embodiment of the present application, a method for automatically managing the capacity of a distributed data analysis engine mainly includes the following steps:

[0042] S1. The management module obtains a number of preset values, determines characteristic values ​​based on the obtained preset values, and divides different data to be stored into data groups with the same number of characteristic values;

[0043] S2. The management module formulates a storage plan table for different data to be stored based on the number of node modules currently in use and the number of all data groups, wherein the number of node modules currently in use corresponds to one of several preset values, and stores all data in several data groups in the corresponding node modules in a secret storage manner according to the storage plan table;

[0044] S3, the management module determines whether the number of currently used node modules needs to be adjusted. If not, repeat this step; if yes, proceed to the next step;

[0045] S4. In the case of reducing the number of node modules currently in use, the management module calculates a first ratio of the characteristic value to the number of node modules after adjustment, calculates a second ratio of the characteristic value to the number of node modules before adjustment, and transfers all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules; in the case of increasing the number of node modules currently in use, the management module calculates a third ratio of the characteristic value to the number of node modules before adjustment, calculates a fourth ratio of the characteristic value to the number of node modules after adjustment, and transfers all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added;

[0046] S5. The extraction module extracts the stored secret data from the corresponding node module, performs recovery processing on the secret data, and the analysis module uses the recovered data for analysis processing.

[0047] Specifically, a distributed data analysis engine usually works on a distributed storage environment. Since the capacity of the distributed storage environment is limited, the capacity should be effectively utilized. For this purpose, S1 to S5 are proposed.

[0048] In S1, the management module obtains several preset values, and determines a characteristic value based on the obtained preset values. It should be noted that the preset values ​​correspond to the number of node modules. For ease of understanding, for example, if the preset values ​​are 3, 6, and 9, then a common multiple of 3, 6, and 9, 18, can be used as a characteristic value. The management module also divides the different data to be stored into data groups with the same number of characteristic values. There is no restriction on the specific division method, as long as the different data to be stored can be evenly divided into data groups with the same number of characteristic values. In S2, the management module formulates a storage plan for different data to be stored based on the number of node modules currently in use and the number of all data groups, wherein the number of node modules currently in use corresponds to one of several preset values. Following the above example, for example, the number of node modules currently in use is 6, and the storage plan records which different data groups each node module should store. Following the above example, for example, the node module with ID 1 to the node module with ID 6 respectively stores any 3 different data groups. The management module stores all data in several data groups in the corresponding node modules in a secret storage manner according to the storage plan. The specific process will be described below. So far, the different data to be stored are stored evenly in each node module, and the capacity can be effectively utilized. In S3, the management module determines whether it is necessary to adjust the number of node modules currently in use. For example, when the total amount of data to be stored increases, in order to effectively utilize the capacity and avoid the problem of centralized storage, the number of node modules must be increased. When the total amount of data to be stored decreases, in order to effectively utilize the capacity and avoid the problem of overly dispersed storage, the number of node modules must be reduced. If no adjustment is required, repeat this step. If adjustment is required, continue to S4. In S4, if the number of node modules currently in use is reduced, the management module calculates a first ratio of the characteristic value to the number of node modules after adjustment, calculates a second ratio of the characteristic value to the number of node modules before adjustment, and transfers all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules. If the number of node modules currently in use is increased, the management module calculates a third ratio of the characteristic value to the number of node modules before adjustment, calculates a fourth ratio of the characteristic value to the number of node modules after adjustment, and transfers all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added. Through this step, regardless of whether the number of node modules needs to be increased or decreased, different node modules can always store data in a balanced manner, thereby effectively utilizing capacity. In addition, when the number of node modules currently in use needs to be adjusted, capacity management is performed in units of data groups, which can improve capacity management efficiency.In S5, the extraction module extracts the required secret data from the corresponding node module and performs recovery processing on the extracted secret data. The specific process will be described below. Then the analysis module uses the recovered data for analysis and processing. The analysis and processing can be implemented using algorithms in the prior art, so it will not be repeated here.

[0049] Further, in S4, when the number of currently used node modules is reduced, the number of node modules after the reduction corresponds to one of several preset values.

[0050] Specifically, if the number of currently used node modules needs to be reduced, the number of node modules after reduction corresponds to one of several preset values. Continuing with the above example, the number of node modules after reduction may be 3. This ensures that capacity management can be performed in units of data groups when the number of currently used node modules needs to be adjusted.

[0051] Further, in S4, when the number of currently used node modules is increased, the increased number of node modules corresponds to one of several preset values.

[0052] Specifically, if the number of currently used node modules needs to be increased, the number of node modules after the increase corresponds to one of several preset values. Continuing with the above example, the number of node modules after the increase may be 9. This ensures that capacity management can be performed in units of data groups when the number of currently used node modules needs to be adjusted.

[0053] Furthermore, the management module stores all data in the plurality of data groups in the corresponding node modules in a secret storage manner, including the following steps:

[0054] S21, for each data in the plurality of data groups, the management module performs confidentiality processing on the data to obtain confidential data;

[0055] S22, with respect to the secret data to be stored corresponding to each data in the plurality of data groups, the management module compares the secret data to be stored with the different secret data stored in the corresponding node module;

[0056] S23. With respect to the secret data to be stored corresponding to each data in several data groups, the management module determines whether there is stored secret data that has the same part as the secret data to be stored. If so, storage processing is performed on different parts of the secret data to be stored; if not, storage processing is performed on the secret data to be stored.

[0057] Specifically, the process of the management module storing all the data in several data groups in the corresponding node modules in a secret storage manner is introduced. In S21, for each data in several data groups, the management module performs secret processing on the data to obtain the secret data. The process of secret processing will be described below. The secret data corresponding to the data is stored, rather than directly storing the data, which can avoid the problem of data leakage. In S22, for the secret data to be stored corresponding to each data in several data groups, the management module compares the secret data to be stored with the different secret data stored in the corresponding node module. The corresponding node module refers to the node module that is about to store the secret data to be stored. The process of comparison processing will be described below. In S23, for the secret data to be stored corresponding to each data in several data groups, the management module determines whether there is a stored secret data that has the same part as the secret data to be stored. If so, the management module performs storage processing on the different parts of the secret data to be stored. If not, the management module directly performs storage processing on the secret data to be stored. Through the above method, when there is stored secret data that has the same part as the secret data to be stored, the amount of data to be stored can be reduced, so that the capacity can be effectively utilized. It should be noted that when storing, different parts of the secret data to be stored can be stored corresponding to the corresponding stored secret data. The purpose of doing this is that when comparison processing is required later, the complete secret data to be stored can be obtained based on the corresponding stored secret data for comparison processing. The corresponding stored secret data can be complete or partial, that is, stored together with other complete secret data.

[0058] Furthermore, the management module performs secret processing on the data to obtain secret data, including the following steps:

[0059] S211, the management module divides the data into a plurality of partial data, and the management module sequentially generates a sequence number for each partial data in the order in which the plurality of partial data are obtained;

[0060] S212, the management module uses all the partial data except the partial data corresponding to the largest sequence number among the plurality of partial data to form intermediate data, and performs a first operation processing on the intermediate data to obtain first operation result data;

[0061] S213, the management module performs a second operation on the first operation result data and the partial data corresponding to the largest sequence number among the plurality of partial data to obtain second operation result data, and generates obfuscated data according to the second operation result data;

[0062] S214, the management module performs a second operation on the obfuscated data and the intermediate data to obtain the first secret data, and performs encryption on the partial data corresponding to the largest sequence number among the partial data to obtain the second secret data, and connects the first secret data and the second secret data to obtain the secret data.

[0063] Specifically, the process of the management module performing secret processing on the data to obtain secret data is introduced. In S211, the management module performs division processing to divide the data into several partial data. It should be noted that the data scale of the several partial data is the same. According to the order of obtaining the several partial data, a sequence number is generated for each partial data in turn. For the sake of ease of understanding, for example, data A is divided in order from left to right to obtain partial data A1, partial data A2, and partial data A3, then the sequence number of partial data A1 to the sequence number of partial data A3 are 1 to 3 in turn. In S212, the management module uses all the other partial data of the several partial data except the partial data corresponding to the largest sequence number to form intermediate data. Following the above example, that is, using partial data A1 to partial data A2 to form intermediate data, the first operation processing is performed on the intermediate data to obtain first operation result data. The data scale of the first operation result data is the same as the data scale of the partial data. The first operation processing can be one of the sha series algorithms, which is mainly selected according to the length of the output data of each algorithm in the sha series algorithm. In S213, the management module performs a second operation on the first operation result data and the partial data corresponding to the largest sequence number among the partial data to obtain the second operation result data. The data size of the second operation result data is the same as the data size of the partial data. In order to facilitate the understanding of the second operation, for example, the second operation can be performed on "0101" and "1111" to obtain "1010". The management module generates obfuscated data according to the second operation result data. The data size of the obfuscated data is the same as the data size of the intermediate data. One of the SHA series algorithms can also be used to generate the obfuscated data. The input data is the second operation result data, and the output data is the obfuscated data. The selection is also mainly based on the length of the output data of each algorithm in the SHA series algorithm. In S214, the management module performs a second operation on the obfuscated data and the intermediate data to obtain the first secret data. The management module performs encryption on the partial data corresponding to the largest sequence number among the partial data to obtain the second secret data. The data size of the second secret data is the same as the data size of the partial data. For example, the AES algorithm can be used for encryption. The first secret data and the second secret data are connected to obtain the secret data. Through the above method, secret data corresponding to the data can be generated to protect the security of the data.

[0064] Furthermore, the management module compares the secret data to be stored with the different secret data stored in the corresponding node modules, including the following steps:

[0065] S221, with respect to each secret data stored in the corresponding node module, the management module performs segmentation processing on the secret data to obtain a first portion of data and a second portion of data, and performs the same segmentation processing on the secret data to be stored to obtain a third portion of data and a fourth portion of data;

[0066] S222. The management module determines whether the first portion of data is identical to the third portion of data. If so, the third portion of data is regarded as the same portion of the secret data to be stored, and the fourth portion of data is regarded as the different portion of the secret data to be stored.

[0067] Specifically, a process is introduced in which the management module compares the secret data to be stored with the different secret data stored in the corresponding node module. In S221, for each secret data stored in the corresponding node module, the management module performs segmentation processing on the secret data. It should be noted that the segmentation process is the same as the above-mentioned division process, so it is not repeated here. Then the management module obtains the first part of data and the second part of data. The position of the first part of data in the stored secret data is equivalent to the position of the above-mentioned intermediate data in the above-mentioned data, and the second part of data is the part other than the first part of data. The management module also performs the same segmentation processing on the secret data to be stored to obtain the third part of data and the fourth part of data. Similarly, the position of the third part of data in the secret data to be stored is equivalent to the position of the above-mentioned intermediate data in the above-mentioned data, and the fourth part of data is the part other than the third part of data. In S222, the management module determines whether the first part of the data is the same as the third part of the data. If they are the same, the third part of the data is regarded as the same part of the secret data to be stored, and the fourth part of the data is regarded as the different part of the secret data to be stored. If they are not the same, it is determined that there is no same part between the secret data to be stored and the stored secret data. It should be noted that only the first part of the data is compared with the third part of the data, and the second part of the data is not compared with the fourth part of the data, because even if the original part of the data corresponding to the second part of the data is the same as the original part of the data corresponding to the fourth part of the data, if the key used to generate the second part of the data is different from the key used to generate the fourth part of the data, then the second part of the data is different from the fourth part of the data.

[0068] Furthermore, the extraction module performs recovery processing on the secret data, including the following steps:

[0069] S51, the extraction module performs segmentation processing on the secret data to obtain partial secret data 1 and partial secret data 2, and continues to perform decryption processing on the partial secret data 2 to obtain decryption result data;

[0070] S52, the extraction module generates mixed data according to the decryption result data, and performs a second operation on the mixed data and part of the secret data to obtain first restored data;

[0071] S53, the extraction module performs a first operation on the first recovery data to obtain intermediate recovery data, and continues to perform a second operation on the intermediate recovery data and the decryption result data to obtain second recovery data, and connects the first recovery data and the second recovery data to obtain the recovered data.

[0072] Specifically, the process of the extraction module performing recovery processing on the extracted secret data is introduced. It should be noted that if the extracted secret data is partial, it needs to be supplemented before the recovery process is performed. In S51, the extraction module performs segmentation processing on the secret data to obtain partial secret data 1 and partial secret data 2. Similarly, the position of partial secret data 1 in the secret data is equivalent to the position of the above-mentioned intermediate data in the above-mentioned data. Partial secret data 2 is the part other than partial secret data 1. The extraction module continues to perform decryption processing on partial secret data 2 to obtain decryption result data. For example, the AES algorithm can be used for decryption processing. In S52, the extraction module generates mixed data based on the decryption result data. It should be noted that the method of generating mixed data from the decryption result data is exactly the same as the method of generating obfuscated data from the second operation result data, that is, the same one in the SHA series of algorithms can be used. The extraction module performs a second operation processing on the mixed data and partial secret data 1 to obtain the first recovery data. In S53, the extraction module performs a first operation on the first recovery data to obtain intermediate recovery data. The specific process is exactly the same as the process of performing the first operation on the intermediate data, that is, the same algorithm in the sha series can be used. The extraction module continues to perform a second operation on the intermediate recovery data and the decryption result data to obtain the second recovery data, thereby connecting the first recovery data and the second recovery data to obtain the recovered data. Through the above method, data for generating secret data can be obtained.

[0073] Furthermore, the extraction module extracts metadata of the secret data while extracting the stored secret data from the corresponding node module. Extracting the metadata of the secret data includes: determining whether the amount of metadata in the memory exceeds a preset threshold value, and if not, extracting the metadata of the secret data; if exceeded, unloading from the memory a number of metadata that have not been accessed in the last 7 days, and continuing to determine whether the amount of metadata in the memory exceeds the preset threshold value, and if not exceeded, extracting the metadata of the secret data; if exceeded, determining a number of metadata using an LRU algorithm, unloading the determined number of metadata from the memory, and extracting the metadata of the secret data.

[0074] Specifically, as the amount of stored data continues to increase, metadata will also continue to grow. Data analysis engines generally need to load metadata to support data analysis and calculations. However, loading too much metadata will lead to slow loading and high memory resource usage, and will also affect subsequent data extraction performance. When extracting the stored secret data from the corresponding node module, the extraction module also extracts the metadata of the secret data. The metadata helps in analysis and processing. When extracting the metadata of the secret data, the method of dynamically loading and unloading metadata can be considered to solve the above problems. The specific approach is: the first step is to determine whether the number of metadata in the memory exceeds the preset number threshold. The number threshold is set according to the actual application scenario. If it does not exceed, the metadata of the secret data is extracted. If it exceeds, several metadata that have not been accessed in the past 7 days are unloaded from the memory. The second step is to continue to determine whether the number of metadata in the memory exceeds the preset number threshold. If it does not exceed, the metadata of the secret data is extracted. If it exceeds, the LRU algorithm is used to determine several metadata. The LRU algorithm determines several metadata that have been recently accessed but have been accessed less frequently. The LRU algorithm is a prior art, so it will not be repeated here. The determined several metadata are unloaded from the memory to extract the metadata of the secret data.

[0075] In addition to the above content, the distributed storage system in this application is also introduced. The system adopts a three-level capacity control strategy of "system-tenant-dataset". System level: A global upper limit is set based on the total amount of physical resources. The limit is triggered when the remaining capacity can only meet 2-3 days of use. The design takes into account the impact of redundancy strategies on available capacity and avoids storage overload through dynamic adjustment. Tenant level: The capacity of a single tenant does not exceed 2 / 3 of the total capacity, and key-value pair configuration files are used to achieve flexible adjustment. This decentralized management mode is consistent with the tenant isolation characteristics of the distributed storage engine, and at the same time draws on the idea of ​​dividing storage space according to business scenarios in table partitioning technology. Model level: Limit the number of data rows to 5 million to control the resource usage of a single analysis model. This fine-grained control is similar to the optimization strategy of distributed databases for column storage, and is also in line with the practice of optimizing storage efficiency through wide table design in the Hadoop ecosystem.

[0076] The system implements a three-level monitoring system. Storage space warning: Check the remaining space of the root directory or cluster every 15 minutes, and the threshold is configurable. This mechanism is similar to the capacity monitoring module of HDFS, but adds fine-grained detection at the tenant level, similar to Kudu's storage layer monitoring design. Data row number warning: Set a row number threshold for the data set. This design refers to the data scale control method of the columnar storage engine, and combines the row-level compression technology of the distributed database. Alarm classification: Distinguish the three-level alarms of system / tenant / dataset, form a three-dimensional control network with three-level capacity restrictions. This hierarchical alarm system has something in common with Elasticsearch's cluster health status monitoring mechanism, but innovatively incorporates business scenarios (tenant directories, single data sets) into trigger conditions.

[0077] In addition to the above content, the distributed data analysis engine in this application is also introduced. In order to improve concurrent processing capabilities and analytical computing performance, distributed data analysis engines generally cache data and some results in memory, but caching will inevitably consume memory resources, so it is very necessary to limit the capacity of the cache.

[0078] For the data cache (DataCache) and result cache (ResultCache), set the cache pools: DataContainer and ResultContainer respectively, and define the interface methods add, remove, and touch to add, delete, and access cache objects. The size of the cache pool is automatically calculated based on the maximum available memory of the application. The specific calculation method is: the data cache defaults to 1 / 3 of the maximum available memory of the application, and the result cache defaults to 1 / 6 of the maximum available memory of the application. In order to flexibly adapt to various scenarios, the size of the cache pool also supports users to customize configuration through system properties. The cache key is designed to include the dimensions and metrics of analytical calculations, and different analytical calculations generate different cache keys.

[0079] The cache pool uses a linked list structure to store cache objects. The linked list supports the LRU algorithm and is therefore defined as LRULink. In the linked list, the most recently accessed object is placed at the end of the linked list, and the least frequently accessed object is placed at the beginning of the linked list. The objects in the linked list all implement the unified interface ISerializable. The main methods in the interface are complete, dispose, empty, and fill. Among them: the complete method is used to add the prepared object to the LRULink, and the dispose method is to delete it from the LRULink and release the object from memory.

[0080] To ensure that the size of the cache pool does not exceed the allocated capacity limit, when the free memory of the cache pool is smaller than the ISerializable object to be added, the swap to disk operation will be triggered. The specific method is: start traversing the object from the beginning of the LRULink, call the empty method of the object to write the data to the disk, and then remove it from the LRULink to release the occupied memory. When the released memory is larger than the object to be added, stop traversing and perform the adding action. When the object swapped to the disk is accessed again, the fill method needs to be called to read the data from the disk. After the reading is completed, it needs to be added to the LRULink again. Before adding, it is also necessary to check whether the free capacity of the cache pool is greater than the size of the object to be added.

[0081] According to another aspect of the embodiment of the present application, refer to Figure 2 As shown, the present application also provides a distributed data analysis engine capacity automatic management system, including a management module, a node module, an extraction module, and an analysis module, to implement a distributed data analysis engine capacity automatic management method described above.

[0082] The functions of each module in the system are as follows:

[0083] A management module is used to obtain a number of preset values, determine characteristic values ​​based on the obtained preset values, and divide different data to be stored into data groups with the same number of characteristic values; at the same time, it is used to formulate a storage plan for different data to be stored based on the number of node modules currently in use and the number of all data groups, the number of node modules currently in use corresponds to one of the several preset values, and all data in the several data groups are stored in the corresponding node modules in a secret storage manner according to the storage plan; and it is used to determine whether the number of node modules currently in use needs to be adjusted; and it is also used to reduce the number of node modules currently in use. , calculating a first ratio of the characteristic value to the number of node modules after adjustment, calculating a second ratio of the characteristic value to the number of node modules before adjustment, transferring all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules, and in the case of increasing the number of node modules currently in use, calculating a third ratio of the characteristic value to the number of node modules before adjustment, calculating a fourth ratio of the characteristic value to the number of node modules after adjustment, transferring all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added;

[0084] A node module, used to store secret data corresponding to the data to be stored;

[0085] An extraction module, used to extract the stored secret data from the corresponding node module and perform recovery processing on the secret data;

[0086] The analysis module is used to perform analysis on the data recovered by the extraction module.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0089] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automatically managing the capacity of a distributed data analysis engine, characterized in that: The method comprises the following steps: S1. The management module obtains a number of preset values, determines characteristic values ​​based on the obtained preset values, and divides different data to be stored into data groups with the same number of characteristic values; S2, the management module formulates a storage plan table for different data to be stored based on the number of node modules currently in use and the number of all data groups, the number of node modules currently in use corresponds to one of several preset values, and stores all data in several data groups in the corresponding node modules in a secret storage manner according to the storage plan table; S3, the management module determines whether the number of currently used node modules needs to be adjusted, and if not, repeats this step, and if yes, proceeds to the next step; S4. In the case of reducing the number of node modules currently in use, the management module calculates a first ratio of the characteristic value to the number of node modules after adjustment, calculates a second ratio of the characteristic value to the number of node modules before adjustment, and transfers all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules; in the case of increasing the number of node modules currently in use, the management module calculates a third ratio of the characteristic value to the number of node modules before adjustment, calculates a fourth ratio of the characteristic value to the number of node modules after adjustment, and transfers all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added; S5. The extraction module extracts the stored secret data from the corresponding node module, performs recovery processing on the secret data, and the analysis module uses the recovered data for analysis processing.

2. The method according to claim 1, characterized in that In the above S4, when the number of currently used node modules is reduced, the number of node modules after reduction corresponds to one of several preset values.

3. The method according to claim 2, characterized in that In the above S4, when the number of currently used node modules is increased, the increased number of node modules corresponds to one of several preset values.

4. The method according to claim 3, characterized in that The management module stores all data in several data groups in corresponding node modules in a secret storage manner, including the following steps: S21, for each data in the plurality of data groups, the management module performs secret processing on the data to obtain secret data; S22, with respect to the secret data to be stored corresponding to each data in the plurality of data groups, the management module compares the secret data to be stored with the different secret data stored in the corresponding node module; S23. With respect to the secret data to be stored corresponding to each data in several data groups, the management module determines whether there is stored secret data that has the same part as the secret data to be stored. If so, storage processing is performed on different parts of the secret data to be stored; if not, storage processing is performed on the secret data to be stored.

5. The method according to claim 4, characterized in that In S21, the management module performs confidentiality processing on the data to obtain confidential data, including the following steps: S211, the management module divides the data into a plurality of partial data, and the management module sequentially generates a sequence number for each partial data in the order in which the plurality of partial data are obtained; S212, the management module uses all the partial data except the partial data corresponding to the largest sequence number among the plurality of partial data to form intermediate data, and performs a first operation processing on the intermediate data to obtain first operation result data; S213, the management module performs a second operation on the first operation result data and the partial data corresponding to the largest sequence number among the plurality of partial data to obtain second operation result data, and generates obfuscated data according to the second operation result data; S214, the management module performs a second operation on the obfuscated data and the intermediate data to obtain the first secret data, and performs encryption on the partial data corresponding to the largest sequence number among the partial data to obtain the second secret data, and connects the first secret data and the second secret data to obtain the secret data.

6. The method according to claim 5, characterized in that In S22, the management module compares the secret data to be stored with the different secret data stored in the corresponding node modules, including the following steps: S221, with respect to each secret data stored in the corresponding node module, the management module performs segmentation processing on the secret data to obtain a first portion of data and a second portion of data, and performs the same segmentation processing on the secret data to be stored to obtain a third portion of data and a fourth portion of data; S222, the management module determines whether the first part of data is the same as the third part of data. If yes, the third part of data is regarded as the same part of the secret data to be stored, and the fourth part of data is regarded as the different part of the secret data to be stored.

7. The method according to claim 6, characterized in that The extraction module performs recovery processing on the secret data, including the following steps: S51, the extraction module performs segmentation processing on the secret data to obtain partial secret data 1 and partial secret data 2, and continues to perform decryption processing on the partial secret data 2 to obtain decryption result data; S52, the extraction module generates mixed data according to the decryption result data, and performs a second operation on the mixed data and part of the secret data to obtain first restored data; S53, the extraction module performs a first operation on the first recovery data to obtain intermediate recovery data, and continues to perform a second operation on the intermediate recovery data and the decryption result data to obtain second recovery data, and connects the first recovery data and the second recovery data to obtain the recovered data.

8. The method according to claim 1, characterized in that The extraction module extracts metadata of the secret data while extracting the stored secret data from the corresponding node module. Extracting metadata of the secret data includes: judging whether the amount of metadata in the memory exceeds a preset threshold value, if not, extracting metadata of the secret data; if exceeded, unloading several metadata that have not been accessed in the last 7 days from the memory, continuing to judge whether the amount of metadata in the memory exceeds a preset threshold value, if not exceeded, extracting metadata of the secret data; if exceeded, determining several metadata using an LRU algorithm, unloading the determined several metadata from the memory, and extracting metadata of the secret data.

9. A capacity automatic management system for a distributed data analysis engine, used to implement the method according to any one of claims 1 to 8, characterized in that: Includes the following modules: A management module is used to obtain a number of preset values, determine characteristic values ​​based on the obtained preset values, and divide different data to be stored into data groups with the same number of characteristic values; at the same time, it is used to formulate a storage plan for different data to be stored based on the number of node modules currently in use and the number of all data groups, the number of node modules currently in use corresponds to one of the several preset values, and all data in the several data groups are stored in the corresponding node modules in a secret storage manner according to the storage plan; and it is used to determine whether the number of node modules currently in use needs to be adjusted; and it is also used to reduce the number of node modules currently in use. , calculating a first ratio of the characteristic value to the number of node modules after adjustment, calculating a second ratio of the characteristic value to the number of node modules before adjustment, transferring all secret data in the data group of the difference between the first ratio and the second ratio from the node modules to be reduced to the corresponding other node modules, and in the case of increasing the number of node modules currently in use, calculating a third ratio of the characteristic value to the number of node modules before adjustment, calculating a fourth ratio of the characteristic value to the number of node modules after adjustment, transferring all secret data in the data group of the difference between the third ratio and the fourth ratio from the node modules currently in use to the corresponding node modules to be added; A node module, used to store secret data corresponding to the data to be stored; An extraction module, used to extract the stored secret data from the corresponding node module and perform recovery processing on the secret data; The analysis module is used to perform analysis on the data recovered by the extraction module.

Images