A distributed storage electronic data processing system

Abstract

The present invention provides an electronic data processing system for distributed storage, which relates to the technical field of distributed storage. The system comprises a data slicing module for slicing, a data analysis module for obtaining data characteristic parameters of sub-data slices, a distributed storage monitoring model for obtaining performance parameters of storage nodes, an allocation scheme optimization module for iteratively optimizing allocation schemes based on a genetic algorithm, and a sub-data slice allocation module for allocating sub-data slices based on an optimal allocation scheme. The present invention processes data characteristic parameters and performance parameters to obtain fitness values, and the fitness values ​​comprehensively evaluate the advantages and disadvantages of allocation schemes of sub-data slices and storage nodes from four levels: storage node error rate, storage node utilization rate, processing time, and matching advantages and disadvantages. Finally, the allocation scheme is iteratively optimized in combination with a genetic algorithm to find the best allocation scheme of sub-data slices and storage nodes, thereby ensuring the high efficiency of subsequent data processing.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to an electronic data processing system with distributed storage. Background Art

[0002] With the reduction in the size of semiconductor devices and the improvement of silicon crystal technology, the manufacturing of today's chips enables more diodes for computing to be integrated on a smaller chip substrate. With the increasing development of mobile Internet, large-scale personal data is created. Based on this social trend, in the era of big data and big computing power, efficient algorithms can be used to make appropriate use of computing resources, so that the computing power of computing clusters for certain problems can be improved by one to two orders of magnitude. For example, in complex and irregular data processing problems, due to the huge size of the data set, the data in the data set cannot be completely concentrated in one storage device, but will be distributed and stored in several or dozens of storage devices. Therefore, once it involves the relationship between data operations, especially the scenario where some of the data needs to be integrated and calculated with other data in the data set after analysis, how to improve the computing load of the cluster, so that the cluster computing power can obtain data processing improvement without changing the hardware resources, and how to better schedule limited computing resources to cope with huge and massive data resources are problems that technicians in this field need to solve.

[0003] In the prior art, a data processing method and data storage system based on computing routing redirection (classification number G06F) with authorization announcement number "CN115268800B" includes the following steps: obtaining a set of distributed storage nodes for data to be calculated in a storage computing task, obtaining data computing weights according to the access delay and computing performance of each distributed storage node, and generating a computing sequence according to the size of the data computing weights; routing the computing task to the distributed storage node according to the distributed storage node corresponding to the computing sequence to execute the computing task, obtaining intermediate data, until all the computing data are calculated, and obtaining a distributed storage node sequence of intermediate data; generating an intermediate data computing sequence from large to small according to the intermediate data computing weights; obtaining a result data distributed storage node sequence according to the intermediate data computing sequence until all the intermediate data are calculated, and completing the processing and storage of the computing data. This method can realize distributed computing of ultra-large-scale data sets.

[0004] However, the existing technology still has major defects. For example, when matching the data to be calculated and the distributed storage nodes, the existing technology only simply considers the computing performance and access delay of the storage nodes, but does not fully consider the matching pros and cons of the data to be calculated and the distributed storage nodes, and thus cannot find the best allocation plan between the data to be calculated and the distributed storage nodes, resulting in insufficient efficiency in subsequent data processing.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The object of the present invention is to provide an electronic data processing system with distributed storage to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A distributed storage electronic data processing system, comprising:

[0009] A data slicing module, used for slicing the electronic data to be processed to obtain a plurality of sub-data slices;

[0010] A data analysis module, used to analyze the sub-data slices to obtain data characteristic parameters of each sub-data slice, wherein the data characteristic parameters include the size of the data slice and the floating-point operation prediction value;

[0011] Distributed storage monitoring module, used to obtain the current performance parameters of each storage node in the distributed storage system, including the number of CPU cores, the operating frequency of each CPU core, memory size, CPU usage, disk I / O performance, error rate, and network bandwidth;

[0012] An allocation scheme optimization module is used to randomly generate multiple allocation schemes to form an initial population with the constraint that the memory size of each storage node after allocation is greater than 0. Based on the number of allocated storage nodes in the allocation scheme, the estimated processing time of each allocated storage node, the performance parameters of each allocated storage node, and the data characteristic parameters of the sub-data slices allocated to the allocated storage node, the fitness value used to characterize the quality of the allocation scheme is calculated. With the minimization of the fitness value as the optimization goal, the initial population is optimized using a genetic algorithm, and the allocation scheme corresponding to the minimum fitness value after the optimization is completed is taken as the optimal allocation scheme;

[0013] The sub-data slice allocation module is used to allocate each sub-data slice after sharding to the corresponding storage node for data processing based on the optimal allocation plan.

[0014] Furthermore, the sharding operation is hash sharding, range sharding or list sharding.

[0015] Furthermore, the method for obtaining the floating-point operation prediction value is as follows:

[0016] 1.1) Obtain multiple sample data slices with known floating-point operation values, and extract the feature data of the sample data slices to form feature vectors. The feature data includes the size of the data slice, the number of fields, statistical features, and the proportion of missing values. The statistical features include the mean, variance, standard deviation, maximum value, and minimum value. The feature vector is represented as follows:

[0017]

[0018] In the formula, The feature vector representing the sample data slice, Respectively represent the data slice size, number of fields, mean, variance, standard deviation, maximum value, minimum value and missing value ratio of the sample data slice;

[0019] 1.2) Based on the linear regression model, a floating-point value prediction model with feature vector as input and floating-point value as output is constructed, and the floating-point value prediction model is trained based on the feature vector and floating-point value of the sample data slice;

[0020] 1.3) Obtain the feature data of the sub-data slice and construct a feature vector, and input the feature vector into the trained floating-point operation value prediction model to obtain the floating-point operation prediction value of the sub-data slice.

[0021] Furthermore, the allocation scheme is expressed as follows:

[0022]

[0023] In the formula, Indicates the allocation scheme, is the i-th gene position in the allocation scheme, which means that in this allocation scheme, the i-th sub-data slice is allocated to the j-th storage node, i is the index of the sub-data slice, and , m is the number of sub-data fragments after sharding, j is the index of the storage node in the distributed storage system, and , n is the number of storage nodes in the distributed storage system.

[0024] Furthermore, the method for obtaining the estimated processing time of the allocated storage node is as follows:

[0025] 2.1) Obtain multiple experimental storage nodes with known performance parameters and multiple experimental data slices with known data characteristic parameters, and allocate no less than one experimental data slice to each experimental storage node for data processing, and count the processing time consumed by each experimental storage node to complete data processing, and construct a data processing feature vector of the experimental storage node based on the performance parameters of the experimental storage node and the data characteristic parameters of the experimental data slice allocated to the experimental storage node. The data processing feature vector is expressed as follows:

[0026]

[0027] In the formula, represents the data processing feature vector of the experimental storage node, represent the performance parameter of the experimental storage node and the data characteristic parameter of the kth experimental data slice allocated to the experimental storage node, respectively, k is the index of the experimental data slice allocated to the experimental storage node, and , K is the number of experimental data slices allocated to the experimental storage node;

[0028]

[0029] In the formula, C represents the number of CPU cores of the experimental storage node, represents the operating frequency of the oth CPU core in the experimental storage node, o is the index of the CPU core in the experimental storage node, and , They represent the memory size, CPU usage, disk I / O performance, error rate, and network bandwidth of the experimental storage node respectively;

[0030]

[0031] In the formula, represent the data slice size and floating point operation prediction value of the kth experimental data slice allocated to the experimental storage node respectively;

[0032] 2.2) Based on the deep neural network, a processing time estimation model is constructed with the input as the data processing feature vector and the output as the processing time. The model is trained based on the data processing feature vector and processing time of the experimental storage node;

[0033] 2.3) Based on the performance parameters of the allocated storage node in the allocation scheme and the data feature parameters of the sub-data slice allocated to the allocated storage node, a data processing feature vector of the allocated storage node is constructed, and the data processing feature vector is input into the trained processing time estimation model to obtain the estimated processing time of the allocated storage node.

[0034] Furthermore, the calculation method of the fitness value is as follows:

[0035] 3.1) Based on the total number of storage nodes in the distributed storage system, the number of allocated storage nodes in the allocation plan, and the estimated processing time of each allocated storage node in the allocation plan, a processing time evaluation coefficient is generated, which is used to characterize the quality of the processing time for data processing based on the allocation plan. The specific calculation formula is as follows:

[0036]

[0037] In the formula, represents the processing time evaluation coefficient, Indicates the processing time evaluation value, They represent the maximum value of the estimated processing time, the average value of the estimated processing time, and the standard deviation of the estimated processing time in the allocation plan, respectively. Indicates the number of allocated storage nodes in the allocation scheme;

[0038] 3.2) Based on the performance parameters of the allocated storage nodes in the allocation scheme and the data characteristic parameters of the sub-data slices allocated to the allocated storage nodes, a matching evaluation index of the allocated storage nodes is generated, which is used to characterize the matching quality between the allocated storage nodes and the sub-data slices allocated to the allocated storage nodes;

[0039] 3.3) Calculate the average matching evaluation index of all allocated storage nodes in the allocation plan, and use the average as the comprehensive matching evaluation index;

[0040] 3.4) Combined with the number of allocated storage nodes in the allocation plan, the error rate of each allocated storage node, the processing time evaluation coefficient corresponding to the allocation plan, and the comprehensive matching evaluation index, the fitness value of the allocation plan is generated. The calculation formula is as follows:

[0041]

[0042] In the formula, represents the mean error rate of all allocated storage nodes in the allocation scheme, It represents the mean error rate of all storage nodes in the distributed storage system, SYD is the fitness value of the allocation scheme, Represents the comprehensive matching evaluation index corresponding to the allocation scheme, , are preset weights, and , The specific value of is determined by the hierarchical analysis method.

[0043] Furthermore, the calculation method of the matching evaluation index is as follows:

[0044] 3.2.1) Based on the memory size of the allocated storage node in the allocation plan and the size of the data slice allocated to the sub-data slice in the allocated storage node, the memory occupancy evaluation index of the allocated storage node is calculated. The calculation formula is as follows:

[0045]

[0046] In the formula, Indicates the memory usage evaluation index of the allocated storage node. Indicates the memory size of the allocated storage node. Indicates the sum of the sizes of all sub-data slices allocated to the allocated storage node;

[0047] 3.2.2) Based on the CPU core operating frequency of the allocated storage node in the allocation plan and the floating point operation prediction value of the sub-data slice allocated to the allocated storage node, the computing performance evaluation index of the allocated storage node is calculated. The calculation formula is as follows:

[0048]

[0049] In the formula, Indicates the computing performance evaluation index of the allocated storage node. Indicates the sum of the operating frequencies of all CPU cores in the allocated storage node. Indicates the sum of the floating-point operation prediction values ​​of all sub-data slices allocated to the allocated storage node;

[0050] 3.2.3) Based on the network bandwidth of the allocated storage node in the allocation plan and the size of the data slice allocated to the sub-data slice in the allocated storage node, the data transmission evaluation index of the allocated storage node is calculated. The calculation formula is as follows:

[0051]

[0052] In the formula, Indicates the data transmission evaluation index of the allocated storage node. Indicates the network bandwidth of the allocated storage node;

[0053] 3.2.4) Based on the memory usage evaluation index, computing performance evaluation index and data transmission evaluation index, the matching evaluation index of the allocated storage node is calculated using the following formula:

[0054]

[0055] In the formula, Indicates the matching evaluation index of the allocated storage node, , , are the weights of memory usage evaluation index, computing performance evaluation index, and data transmission evaluation index in the calculation of matching evaluation index, respectively, and , , The specific value of is determined by the hierarchical analysis method.

[0056] Furthermore, the process of using the genetic algorithm to optimize the initial population is as follows: each allocation scheme in the initial population is regarded as an individual, and each individual is sorted from small to large according to the fitness value, and the top 50% of the individuals are selected as parents for crossover and mutation operations to generate new individuals. The new individuals and parents are used as the new population, and the fitness value of each individual in the new population is calculated, and the selection, crossover and mutation operations are repeated until the optimization ends when the predetermined number of iterations is reached, and the allocation scheme corresponding to the minimum fitness value after the optimization is completed is taken as the optimal allocation scheme.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The distributed storage electronic data processing system of the present invention first obtains data characteristic parameters of sub-data slices and performance parameters of storage nodes through a data analysis module and a distributed storage monitoring module, and then processes the data characteristic parameters and performance parameters to obtain fitness values. The fitness values ​​comprehensively and fully evaluate the advantages and disadvantages of allocation schemes of sub-data slices and storage nodes from four levels: storage node error rate, storage node utilization rate, processing time, and matching advantages and disadvantages. Finally, the allocation schemes are iteratively optimized in combination with a genetic algorithm to find the best allocation scheme for sub-data slices and storage nodes, thereby ensuring the high efficiency of subsequent data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a module unit diagram of the overall system of the present invention. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0061] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0062] Example:

[0063] See also Figure 1 The present invention provides a distributed storage electronic data processing system, comprising:

[0064] A data slicing module, used for slicing the electronic data to be processed to obtain a plurality of sub-data slices;

[0065] The sharding operation may adopt conventional sharding operations such as hash sharding, range sharding or list sharding, which may be selected according to the actual situation and are not limited here.

[0066] It should be noted that the specific operation of hash sharding is: using a hash function to perform a hash operation on the primary key of the electronic data to be processed, so as to evenly divide the electronic data to be processed into several sub-data slices. This sharding method can efficiently disperse data and reduce the occurrence of hot data (i.e., even sharding), but it has poor support for range queries;

[0067] It should be noted that the specific operation of range sharding is: according to a specific attribute of the electronic data to be processed (such as timestamp, ID, etc.), the electronic data to be processed is sharded by range. If the electronic data to be processed is a user data set, the sharding operation is performed based on the range of user IDs, specifically, data with user IDs within 1-1000 are divided into a sub-data slice, data with user IDs within 1001-2000 are divided into a sub-data slice, and so on. This sharding method is suitable for range queries and can quickly locate related data, but it may cause the problem of too large or too small data volume in some sub-data slices, resulting in uneven sharding;

[0068] It should be noted that the specific operation of list sharding is: manually defining sharding rules to divide the electronic data to be processed into several sub-data slices. This sharding method is suitable for electronic data to be processed that is small in quantity and has known rules.

[0069] A data analysis module, used to analyze the sub-data slices to obtain data characteristic parameters of each sub-data slice, wherein the data characteristic parameters include the size of the data slice and the floating-point operation prediction value;

[0070] The data slice size is the storage space required by the data slice in the storage system, which is generally in bytes and can be directly obtained through the SQL query function;

[0071] The floating point operation prediction value is the predicted floating point operation quantity before data processing is performed on the sub-data slice, and the method for obtaining the floating point operation prediction value is as follows:

[0072] 1.1) Obtain multiple sample data slices with known floating-point operation values, and extract the feature data of the sample data slices to form feature vectors. The feature data includes the size of the data slice, the number of fields, statistical features, and the proportion of missing values. The statistical features include the mean, variance, standard deviation, maximum value, and minimum value. The feature vector is represented as follows:

[0073]

[0074] In the formula, The feature vector representing the sample data slice, Respectively represent the data slice size, number of fields, mean, variance, standard deviation, maximum value, minimum value and missing value ratio of the sample data slice;

[0075] It should be noted that the number of fields is the number of columns in the data slice. The size of the data slice, the number of fields, the statistical characteristics, and the proportion of missing values ​​are all conventional feature data, which can be directly obtained by extracting features from the sample data slice through the Python library. This is a prior art and will not be described in detail here. The floating-point operation value of the sample data slice can be obtained through data processing libraries such as NumPy and TensorFlow. For example, TensorFlow provides a tool for monitoring the operation process, which includes statistics on the number of floating-point operations.

[0076] 1.2) Based on the linear regression model, a floating-point value prediction model with feature vector as input and floating-point value as output is constructed. The floating-point value prediction model is trained based on the feature vector and floating-point value of the sample data slice. The specific training method is as follows:

[0077] The floating-point operation values ​​of each sample data slice are marked on the corresponding feature vectors one by one, and the marked feature vectors are used to form a data set. The data set is randomly divided into a training set and a test set in a ratio of 80 to 20. The training set is used to train the floating-point operation value prediction model, and the mean square error is used as the loss function during the training process. The model parameters are iteratively optimized by minimizing the difference between the model prediction value and the actual value, and cross-validation is used to select the optimal model hyperparameters to avoid overfitting. After reaching the predetermined number of iterations, the floating-point operation value prediction model is tested using the test set. Specifically, the root mean square error and mean absolute error can be used as evaluation indicators. If the evaluation indicators meet the requirements, the training is considered to be completed, otherwise, the training is repeated until the evaluation indicators meet the requirements. The specific requirements are set according to the actual situation. If the actual value is between 0 and 100, the root mean square error and mean absolute error values ​​less than 10 can be considered to meet the evaluation indicators.

[0078] 1.3) Obtain the feature data of the sub-data slice and construct a feature vector, and input the feature vector into the trained floating-point operation value prediction model to obtain the floating-point operation prediction value of the sub-data slice;

[0079] The method for obtaining the feature vector of the sub-data slice is the same as that in step 1.1) above, and will not be described in detail here;

[0080] Furthermore, in order to improve the convergence speed and performance of the floating-point operation value prediction model, the data belonging to the same category in the feature data of the data set and the sub-data slice (for example, the data slice size of the sub-data slice and the data slice size of the sample data slice in the data set belong to the same category of data) can be subjected to maximum-minimum normalization processing together to unify the dimensions of various types of data, thereby improving the convergence speed and performance of the subsequent floating-point operation value prediction model;

[0081] It should be noted that when the floating-point operation value prediction model is established to predict the floating-point operation prediction value of each sub-data slice, different operation types need to be modeled separately. For example, a floating-point operation value prediction model for query operations is established, an insert / update floating-point operation value prediction model is established for insert / update operations, and a calculation floating-point operation value prediction model is established for calculation operations. Then, according to the type of data processing operation that needs to be performed on each sub-data slice, its feature data is input into the corresponding floating-point operation value prediction model to obtain accurate floating-point operation prediction values.

[0082] Distributed storage monitoring module, used to obtain the current performance parameters of each storage node in the distributed storage system, including the number of CPU cores, the operating frequency of each CPU core, memory size, CPU usage, disk I / O performance, error rate, and network bandwidth;

[0083] The number of CPU cores is the number of CPU cores in the storage node, that is, the number of independent working units in the storage node. The working frequency is the number of instructions that the CPU core can execute per second. The higher the working frequency, the stronger the computing power of the CPU core. It can be obtained through the tools and commands provided by the storage node, such as through the task manager.

[0084] The memory size is the available memory capacity of the storage node, which affects the data cache and processing capabilities. The method of obtaining it is: use system commands (such as Linux's free command) or monitoring tools (such as Prometheus, Grafana) to obtain the occupied memory capacity and total memory capacity of the storage node at the current moment. The difference between the total memory capacity and the occupied memory capacity is the memory size.

[0085] The CPU usage is the percentage of CPU resources used by the storage node, which reflects the computing load of the storage node. The CPU usage of the storage node at the current moment can be obtained by using system commands (such as Linux top or htop commands).

[0086] Disk I / O performance refers to the disk read and write speed of the storage node, usually expressed in IOPS (input and output operations per second). It affects the read and write efficiency of data. The method of obtaining it is as follows: Use tools (such as iostat, dd, fio, etc.) to test the storage node to obtain the disk I / O performance of the storage node at the current moment;

[0087] The network bandwidth is the network transmission speed of the storage node, which affects the data transmission efficiency between nodes. The acquisition method is: obtain the network bandwidth of the storage node at the current moment through network monitoring tools (such as iftop, nload, iperf, etc.);

[0088] The error rate is the ratio of errors that occur during the data reading and writing process of the storage node. An excessively high error rate indicates that the storage node has a hardware failure or a system configuration problem. The error rate is obtained by monitoring and counting the system log (such as dmesg) and the application error log to obtain the total amount of data and the amount of erroneous data in the reading and writing process of the storage node in the time period before the current moment (such as three minutes, five minutes, etc., without limitation). The ratio of the amount of erroneous data to the total amount is the error rate.

[0089] The allocation scheme optimization module is used to randomly generate multiple allocation schemes to form an initial population based on the constraint that the memory size of each storage node after allocation is greater than 0. Based on the number of allocated storage nodes in the allocation scheme, the estimated processing time of each allocated storage node, the performance parameters of each allocated storage node, and the data characteristic parameters of the sub-data slices allocated to the allocated storage nodes, the fitness value used to characterize the quality of the allocation scheme is calculated, and the larger the fitness value, the worse the quality of the allocation scheme. The initial population is optimized using a genetic algorithm with the fitness value minimized as the optimization goal, and the allocation scheme corresponding to the minimum fitness value after the optimization is completed is taken as the optimal allocation scheme;

[0090] The specific judgment method for determining whether the memory size of each storage node after allocation is greater than 0 is as follows: for any storage node, the total size of the data slices planned to be allocated to the storage node is less than the current memory size of the storage node. This constraint condition is set to avoid the problem of memory exhaustion of the storage node, thereby avoiding data loss or even system crash caused by failure to write sub-data slices;

[0091] The allocation scheme is specifically: each sub-data slice is randomly allocated to any storage node in the distributed storage system, that is, a sub-data slice can and can only be allocated to one storage node, and a storage node can be allocated to 0, 1, or more than 1 sub-data slices. The allocation scheme is specifically expressed in the following form:

[0092]

[0093] In the formula, Indicates the allocation scheme, is the i-th gene position in the allocation scheme, which means that in this allocation scheme, the i-th sub-data slice is allocated to the j-th storage node, i is the index of the sub-data slice, and , m is the number of sub-data fragments after sharding, j is the index of the storage node in the distributed storage system, and , n is the number of storage nodes in the distributed storage system, and the storage node index j corresponding to each gene bit in the above allocation scheme may be different;

[0094] The allocated storage node is the storage node allocated to the sub-data slice in the allocation plan. The estimated processing time of the allocated storage node is the time consumed by the allocated storage node to complete data processing of all the sub-data slices allocated to itself. The method for obtaining the estimated processing time is as follows:

[0095] 2.1) Obtain multiple experimental storage nodes with known performance parameters and multiple experimental data slices with known data characteristic parameters, and allocate no less than one experimental data slice to each experimental storage node for data processing, and count the processing time consumed by each experimental storage node to complete data processing, and construct a data processing feature vector of the experimental storage node based on the performance parameters of the experimental storage node and the data characteristic parameters of the experimental data slice allocated to the experimental storage node. The data processing feature vector is expressed as follows:

[0096]

[0097] In the formula, represents the data processing feature vector of the experimental storage node, represent the performance parameter of the experimental storage node and the data characteristic parameter of the kth experimental data slice allocated to the experimental storage node, respectively, k is the index of the experimental data slice allocated to the experimental storage node, and , K is the number of experimental data slices allocated to the experimental storage node;

[0098]

[0099] In the formula, C represents the number of CPU cores of the experimental storage node, represents the operating frequency of the oth CPU core in the experimental storage node, o is the index of the CPU core in the experimental storage node, and , They represent the memory size, CPU usage, disk I / O performance, error rate, and network bandwidth of the experimental storage node respectively;

[0100]

[0101] In the formula, represent the data slice size and floating point operation prediction value of the kth experimental data slice allocated to the experimental storage node respectively;

[0102] It should be noted that the performance parameters of the experimental storage node and the data characteristic parameters of the experimental data slice can be obtained by the above method, which will not be described in detail here;

[0103] 2.2) Based on the deep neural network, a processing time estimation model is constructed with the input as the data processing feature vector and the output as the processing time. The model is trained based on the data processing feature vector and processing time of the experimental storage node. The specific training method is as follows:

[0104] The processing time is uniformly expressed in seconds. Each processing time is marked on the corresponding data processing feature vector one by one. The marked data processing feature vectors form a data set, and the data set is randomly divided into a training set and a test set in a ratio of 80:20. The training set is used to train the processing time estimation model. During the training process, the mean square error is used as the loss function. The model parameters are iteratively optimized by minimizing the difference between the model prediction value and the actual value, and cross-validation is used to select the optimal model hyperparameters to avoid overfitting. After reaching the predetermined number of iterations, the processing time estimation model is tested using the test set. If the test requirements are met, the training is considered to be completed. Otherwise, the training is repeated until the test requirements are met. The specific test requirements can be set to the absolute deviation between the model prediction value and the actual value not higher than 0.01 second, 0.1 second, 0.5 second, 1 second, etc., and there is no restriction here.

[0105] 2.3) Based on the performance parameters of the allocated storage node in the allocation scheme and the data feature parameters of the sub-data slice allocated to the allocated storage node, a data processing feature vector of the allocated storage node is constructed, and the data processing feature vector is input into the trained processing time estimation model to obtain the estimated processing time of the allocated storage node;

[0106] The calculation method of the fitness value is as follows:

[0107] 3.1) Based on the total number of storage nodes in the distributed storage system, the number of allocated storage nodes in the allocation plan, and the estimated processing time of each allocated storage node in the allocation plan, a processing time evaluation coefficient is generated, which is used to characterize the quality of the processing time for data processing based on the allocation plan. The specific calculation formula is as follows:

[0108]

[0109] In the formula, Indicates the processing time evaluation coefficient, which evaluates the processing time of the allocation scheme from two aspects: the processing time of the allocation scheme and the utilization degree of the distributed storage system. The larger the processing time evaluation coefficient, the worse the processing time of data processing based on the allocation scheme, and the larger the optimization space of the allocation scheme;

[0110] In the formula, It indicates the evaluation value of processing time. The larger the evaluation value of processing time is, the worse the processing time of data processing based on the allocation scheme is, that is, the larger the optimization space of the allocation scheme is.

[0111] In the formula, They represent the maximum value of the estimated processing time, the average value of the estimated processing time, and the standard deviation of the estimated processing time in the allocation scheme, respectively. All three can be obtained by processing the estimated processing time of each allocated storage node in the allocation scheme based on conventional data processing methods, which will not be elaborated here;

[0112] It should be noted that the larger the maximum value of the estimated processing time is, the longer it takes for the distributed storage system to complete data processing, that is, the worse the processing time of data processing based on the allocation scheme is, and the larger the optimization space of the allocation scheme is;

[0113] It should be noted that the larger the ratio of the estimated processing time standard deviation to the estimated processing time average value, the greater the difference in the time consumed by each allocated storage node for data processing when data processing is performed based on the allocation scheme, and the longer the waiting time for the allocated storage node that completes data processing first to wait for the allocated storage node that completes data processing later to aggregate data together, indicating that the consistency of the time for each allocated storage node to complete data processing is worse, the worse the processing time quality of data processing based on the allocation scheme is, and the greater the optimization space of the allocation scheme is;

[0114] It should be noted that, based on the smaller maximum estimated processing time, even if the ratio No matter how large the value is, the data processing can still be completed quickly based on the allocation scheme, and based on the large maximum estimated processing time, even if the ratio No matter how small the value is, the time consumed to complete data processing based on the allocation scheme is still long, that is, the maximum estimated processing time can fundamentally reflect the pros and cons of the processing time for data processing based on the allocation scheme. It indirectly reflects the quality of the processing time for data processing based on the allocation scheme. Therefore, when using the maximum estimated processing time to characterize the quality of the processing time, the ratio is used. To amplify the impact of the maximum estimated processing time, As the base, the ratio As an index, The processing time evaluation value is calculated in the form of 1. The setting of 1 avoids the problem of the base being less than 1. This nonlinearly characterizes the pros and cons of the processing time for data processing based on the allocation scheme. The larger the maximum value of the estimated processing time, the higher the ratio. The larger it is, the worse the processing time of data processing based on the allocation scheme is. The larger the optimization space of the allocation scheme is, the larger the processing time evaluation value is.

[0115] In the formula, Indicates the number of allocated storage nodes in the allocation scheme, the ratio The smaller the value, the less the utilization of the distributed storage system is when processing data based on the allocation scheme, and the larger the optimization space of the allocation scheme is. Therefore, the ratio is used. The processing time evaluation value is modified to obtain the processing time evaluation coefficient, so that the processing time evaluation coefficient evaluates the processing time of the allocation scheme from two aspects: processing time and utilization of the distributed storage system. The larger the processing time evaluation value, the higher the ratio. The smaller it is, the worse the processing time of data processing based on the allocation scheme is. The larger the optimization space of the allocation scheme is, the larger the processing time evaluation coefficient is.

[0116] 3.2) Based on the performance parameters of the allocated storage node in the allocation scheme and the data characteristic parameters of the sub-data slice allocated to the allocated storage node, a matching evaluation index of the allocated storage node is generated, which is used to characterize the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node, as follows:

[0117] 3.2.1) Based on the memory size of the allocated storage node in the allocation plan and the size of the data slice allocated to the sub-data slice in the allocated storage node, the memory occupancy evaluation index of the allocated storage node is calculated. The calculation formula is as follows:

[0118]

[0119] In the formula, Indicates the memory occupancy evaluation index of the allocated storage node, which is used to evaluate the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node from the memory occupancy level, and the larger the memory occupancy evaluation index, the worse the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node;

[0120] In the formula, Indicates the memory size of the allocated storage node. Indicates the sum of the sizes of all sub-slices allocated to the allocated storage node, the ratio The larger the value is, the smaller the remaining available memory of the allocated storage node is after allocating the sub-data slice to the allocated storage node, which means the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node is worse.

[0121] 3.2.2) Based on the CPU core operating frequency of the allocated storage node in the allocation plan and the floating point operation prediction value of the sub-data slice allocated to the allocated storage node, the computing performance evaluation index of the allocated storage node is calculated. The calculation formula is as follows:

[0122]

[0123] In the formula, Indicates the computing performance evaluation index of the allocated storage node, which is used to evaluate the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node in terms of computing capability matching. The larger the computing performance evaluation index, the worse the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node.

[0124] In the formula, Indicates the sum of the operating frequencies of all CPU cores in the allocated storage node. Indicates the sum of the floating-point operation prediction values ​​of all sub-data slices allocated to the allocated storage node, the ratio The larger the value is, the more difficult it is for the allocated storage node to quickly process the allocated sub-data slice, that is, the worse the ability of the allocated storage node to process the allocated sub-data slice is, and the worse the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node is.

[0125] 3.2.3) Based on the network bandwidth of the allocated storage node in the allocation plan and the size of the data slice allocated to the sub-data slice in the allocated storage node, the data transmission evaluation index of the allocated storage node is calculated. The calculation formula is as follows:

[0126]

[0127] In the formula, Indicates the data transmission evaluation index of the allocated storage node, which is used to evaluate the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node from the data transmission level, and the larger the data transmission evaluation index, the worse the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node;

[0128] In the formula, Indicates the network bandwidth of the allocated storage node, ratio The larger the value is, the less efficient the transmission of the allocated sub-data slice to the allocated storage node is, that is, the worse the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node is.

[0129] 3.2.4) Based on the memory usage evaluation index, computing performance evaluation index and data transmission evaluation index, the matching evaluation index of the allocated storage node is calculated using the following formula:

[0130]

[0131] In the formula, Indicates the matching evaluation index of the allocated storage node, which combines the memory occupancy evaluation index, the computing performance evaluation index and the data transmission evaluation index to comprehensively evaluate the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node from the three levels of memory occupancy, computing performance and data transmission. The larger the memory occupancy evaluation index, the computing performance evaluation index and the data transmission evaluation index, the worse the matching quality between the allocated storage node and the sub-data slice allocated to the allocated storage node, and the larger the matching evaluation index;

[0132] In the formula, , , are the weights of memory usage evaluation index, computing performance evaluation index, and data transmission evaluation index in the calculation of matching evaluation index, respectively, and , , The specific value of is determined by the hierarchical analysis method, and the specific logic is as follows:

[0133] The nine-scale method is used to determine the relative importance of the memory usage evaluation index, the computing performance evaluation index, and the data transmission evaluation index, and a judgment matrix is ​​constructed, in which the index of the memory usage evaluation index is marked as 1, the index of the computing performance evaluation index is marked as 2, and the index of the data transmission evaluation index is marked as 3. The constructed judgment matrix is:

[0134]

[0135] Among them, f and v both represent the index of the evaluation index, and , , which indicates the importance of the evaluation index with index f to the matching evaluation index relative to the evaluation index with index v. The specific value is determined by relevant experts using a 1-9 scoring method. Indicates that the evaluation index with index f is more important for matching the evaluation index than the evaluation index with index v. Indicates that the evaluation index of index f is extremely unimportant to the matching evaluation index compared to the evaluation index of index v;

[0136] Divide each element value in the judgment matrix by the sum of its columns to obtain a normalized judgment matrix, calculate the mean of the element values ​​in each row of the normalized judgment matrix, and use the mean of the element values ​​in the first row as the proportional coefficient of the memory usage evaluation index, the mean of the element values ​​in the second row as the proportional coefficient of the computing performance evaluation index, and the mean of the element values ​​in the third row as the proportional coefficient of the data transmission evaluation index. With the constraint that the sum of the scaled values ​​is equal to 1, scale the five proportional coefficients in equal proportions, and use the scaled values ​​as the weights of the corresponding evaluation indexes.

[0137] 3.3) Calculate the average matching evaluation index of all allocated storage nodes in the allocation plan, and use the average as the comprehensive matching evaluation index. The comprehensive matching evaluation index is used to evaluate the overall quality of the matching plan. The larger the comprehensive matching evaluation index, the worse the matching plan.

[0138] 3.4) Combined with the number of allocated storage nodes in the allocation plan, the error rate of each allocated storage node, the processing time evaluation coefficient corresponding to the allocation plan, and the comprehensive matching evaluation index, the fitness value of the allocation plan is generated. The calculation formula is as follows:

[0139]

[0140] In the formula, represents the mean error rate of all allocated storage nodes in the allocation scheme, It represents the average error rate of all storage nodes in the distributed storage system. Both can be obtained by processing the current error rate of the storage nodes in the distributed storage system through conventional data processing means, which will not be elaborated here. Indicates the comprehensive matching evaluation index corresponding to the allocation scheme;

[0141] In the formula, SYD is the fitness value of the allocation scheme, which comprehensively evaluates the pros and cons of the allocation scheme from four aspects: error rate, storage node utilization rate, processing time, and matching pros and cons. The worse the pros and cons of the allocation scheme, the greater the fitness value.

[0142] It should be noted that the processing time evaluation coefficient directly reflects the quality of the allocation scheme from the processing time level, and the comprehensive matching evaluation index directly reflects the quality of the allocation scheme from the matching fitness level of the allocated storage nodes and sub-data slices, that is, both fundamentally reflect the quality of the allocation scheme. Therefore, the processing time evaluation coefficient and the comprehensive matching evaluation index are used to construct the main part of the fitness value. The larger the processing time evaluation coefficient and the comprehensive matching evaluation index, the worse the quality of the allocation scheme, and the larger the main part of the fitness value.

[0143] In the formula, , are the weights of the processing time evaluation coefficient and the comprehensive matching evaluation index in the calculation of the main part of the fitness value, respectively, and , The specific value of is determined by the hierarchical analysis method. Please refer to the above for details and will not be elaborated here;

[0144] It should be noted that the ratio The larger the ratio is, the more storage nodes are needed when processing data according to the allocation scheme, which is more likely to cause unnecessary waste of resources, greatly increase the difficulty of management and monitoring, and cause network congestion. The larger the value, the worse the allocation scheme is. Therefore, use The main part of the fitness value is corrected once so that the corrected fitness value takes the storage node usage into consideration;

[0145] It should be noted that the ratio The larger the value, the greater the error rate of the storage nodes (i.e., the allocated storage nodes) used by the allocation scheme, the more likely it is to have read and write errors, and the more likely it is to cause errors in the subsequent data processing process. In other words, the worse the allocated storage nodes selected by the allocation scheme, the worse the allocation scheme is. Therefore, use The main part of the fitness value is corrected twice so that the corrected fitness value takes the storage node error rate into consideration.

[0146] Among them, the process of using genetic algorithm to optimize the initial population is as follows: each allocation scheme in the initial population is regarded as an individual, and each individual is sorted from small to large according to the fitness value, and the first 50% of the individuals are selected as parents for crossover and mutation operations to generate new individuals. The new individuals and parents are used as the new population, and the fitness value of each individual in the new population is calculated. The selection, crossover and mutation operations are repeated until the optimization ends when the predetermined number of iterations is reached. The allocation scheme corresponding to the minimum fitness value after the optimization is completed is used as the optimal allocation scheme. The crossover operation is to exchange part of the gene segments of two individuals. The mutation operation is to randomly change the index of the storage node in the gene position within the range of positive integers 1-n. The number of iterations is set according to the actual situation, such as 50 rounds, 75 rounds, 100 rounds, etc., which are not limited here.

[0147] The sub-data slice allocation module is used to allocate each sub-data slice after sharding to the corresponding storage node for data processing based on the optimal allocation plan. The specific allocation method is: based on the optimal allocation plan, a mapping relationship between each sub-data slice index and the storage node index is established, and then a connection between the sub-data slice and the corresponding storage node is established according to the mapping relationship, so as to route each sub-data slice to the corresponding storage node. This is the existing technology and will not be elaborated here.

[0148] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0149] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A distributed storage electronic data processing system, characterized in that: include: A data slicing module, used for slicing the electronic data to be processed to obtain a plurality of sub-data slices; A data analysis module, used to analyze the sub-data slices to obtain data characteristic parameters of each sub-data slice, wherein the data characteristic parameters include the size of the data slice and the floating-point operation prediction value; Distributed storage monitoring module, used to obtain the current performance parameters of each storage node in the distributed storage system, including the number of CPU cores, the operating frequency of each CPU core, memory size, CPU usage, disk I / O performance, error rate, and network bandwidth; An allocation scheme optimization module is used to randomly generate multiple allocation schemes to form an initial population with the constraint that the memory size of each storage node after allocation is greater than 0. Based on the number of allocated storage nodes in the allocation scheme, the estimated processing time of each allocated storage node, the performance parameters of each allocated storage node, and the data characteristic parameters of the sub-data slices allocated to the allocated storage node, the fitness value used to characterize the quality of the allocation scheme is calculated. With the minimization of the fitness value as the optimization goal, the initial population is optimized using a genetic algorithm, and the allocation scheme corresponding to the minimum fitness value after the optimization is completed is taken as the optimal allocation scheme; A sub-data slice allocation module is used to allocate each sub-data slice after sharding to a corresponding storage node for data processing based on an optimal allocation scheme; The fitness value is calculated as follows: 3.1) Based on the total number of storage nodes in the distributed storage system, the number of allocated storage nodes in the allocation plan, and the estimated processing time of each allocated storage node in the allocation plan, a processing time evaluation coefficient is generated. The calculation formula is as follows: In the formula, represents the processing time evaluation coefficient, Indicates the processing time evaluation value, They represent the maximum value of the estimated processing time, the average value of the estimated processing time, and the standard deviation of the estimated processing time in the allocation plan, respectively. Indicates the number of allocated storage nodes in the allocation scheme; 3.2) Based on the performance parameters of the allocated storage nodes in the allocation scheme and the data characteristic parameters of the sub-data slices allocated to the allocated storage nodes, a matching evaluation index of the allocated storage nodes is generated, which is used to characterize the matching quality between the allocated storage nodes and the sub-data slices allocated to the allocated storage nodes; 3.3) Calculate the average matching evaluation index of all allocated storage nodes in the allocation plan, and use the average matching evaluation index as the comprehensive matching evaluation index; 3.4) Combined with the number of allocated storage nodes in the allocation plan, the error rate of each allocated storage node, the processing time evaluation coefficient corresponding to the allocation plan, and the comprehensive matching evaluation index, the fitness value of the allocation plan is generated. The calculation formula is as follows: In the formula, represents the mean error rate of all allocated storage nodes in the allocation scheme, It represents the mean error rate of all storage nodes in the distributed storage system, SYD is the fitness value of the allocation scheme, Represents the comprehensive matching evaluation index corresponding to the allocation scheme, , are preset weights, and , The specific value of is determined by the hierarchical analysis method, and n is the number of storage nodes in the distributed storage system.

2. The distributed storage electronic data processing system according to claim 1, characterized in that: The sharding operation is hash sharding, range sharding or list sharding.

3. The distributed storage electronic data processing system according to claim 1, characterized in that: The method for obtaining the floating-point operation prediction value is as follows: 1.1) Obtain multiple sample data slices with known floating-point operation values, and extract the feature data of the sample data slices to form feature vectors. It includes the data slice size, number of fields, statistical characteristics and missing value ratio. The statistical characteristics include mean, variance, standard deviation, maximum value and minimum value. The feature vector is expressed as follows: In the formula, The feature vector representing the sample data slice, Respectively represent the data slice size, number of fields, mean, variance, standard deviation, maximum value, minimum value and missing value ratio of the sample data slice; 1.2) Based on the linear regression model, a floating-point value prediction model with feature vector as input and floating-point value as output is constructed, and the floating-point value prediction model is trained based on the feature vector and floating-point value of the sample data slice; 1.3) Obtain the feature data of the sub-data slice and construct a feature vector, and input the feature vector into the trained floating-point operation value prediction model to obtain the floating-point operation prediction value of the sub-data slice.

4. The distributed storage electronic data processing system according to claim 1, characterized in that: The allocation scheme is expressed as follows: In the formula, Indicates the allocation scheme, is the gene bit corresponding to the i-th sub-data slice in the allocation scheme, which means that in the allocation scheme, the i-th sub-data slice is allocated to the j-th storage node, i is the index of the sub-data slice, and , m is the number of sub-data fragments after sharding, j is the index of the storage node in the distributed storage system, and , n is the number of storage nodes in the distributed storage system.

5. The distributed storage electronic data processing system according to claim 1, characterized in that: The method for obtaining the estimated processing time of the allocated storage node is as follows: 2.1) Obtain multiple experimental storage nodes with known performance parameters and multiple experimental data slices with known data characteristic parameters, and allocate no less than one experimental data slice to each experimental storage node for data processing, and count the processing time consumed by each experimental storage node to complete data processing, and construct a data processing feature vector of the experimental storage node based on the performance parameters of the experimental storage node and the data characteristic parameters of the experimental data slice allocated to the experimental storage node. The data processing feature vector is expressed as follows: In the formula, represents the data processing feature vector of the experimental storage node, represent the performance parameter of the experimental storage node and the data characteristic parameter of the kth experimental data slice allocated to the experimental storage node, respectively, k is the index of the experimental data slice allocated to the experimental storage node, and , K is the number of experimental data slices allocated to the experimental storage node; In the formula, C represents the number of CPU cores of the experimental storage node, represents the operating frequency of the oth CPU core in the experimental storage node, o is the index of the CPU core in the experimental storage node, and , They represent the memory size, CPU usage, disk I / O performance, error rate, and network bandwidth of the experimental storage node respectively; In the formula, represent the data slice size and floating point operation prediction value of the kth experimental data slice allocated to the experimental storage node respectively; 2.2) Based on the deep neural network, a processing time estimation model is constructed with the input as the data processing feature vector and the output as the processing time. The model is trained based on the data processing feature vector and processing time of the experimental storage node; 2.3) Based on the performance parameters of the allocated storage node in the allocation scheme and the data feature parameters of the sub-data slice allocated to the allocated storage node, a data processing feature vector of the allocated storage node is constructed, and the data processing feature vector is input into the trained processing time estimation model to obtain the estimated processing time of the allocated storage node.

6. The distributed storage electronic data processing system according to claim 1, characterized in that: The calculation method of the matching evaluation index is as follows: 3.2.1) Based on the memory size of the allocated storage node in the allocation plan and the size of the data slice allocated to the sub-data slice in the allocated storage node, the memory occupancy evaluation index of the allocated storage node is calculated. The calculation formula is as follows: In the formula, Indicates the memory usage evaluation index of the allocated storage node. Indicates the memory size of the allocated storage node. Indicates the sum of the sizes of all sub-data slices allocated to the allocated storage node; 3.2.2) Based on the CPU core operating frequency of the allocated storage node in the allocation plan and the floating point operation prediction value of the sub-data slice allocated to the allocated storage node, the computing performance evaluation index of the allocated storage node is calculated. The calculation formula is as follows: In the formula, Indicates the computing performance evaluation index of the allocated storage node. Indicates the sum of the operating frequencies of all CPU cores in the allocated storage node. Indicates the sum of the floating-point operation prediction values ​​of all sub-data slices allocated to the allocated storage node; 3.2.3) Based on the network bandwidth of the allocated storage node in the allocation plan and the size of the data slice allocated to the sub-data slice in the allocated storage node, the data transmission evaluation index of the allocated storage node is calculated. The calculation formula is as follows: In the formula, Indicates the data transmission evaluation index of the allocated storage node. Indicates the network bandwidth of the allocated storage node; 3.2.4) Based on the memory usage evaluation index, computing performance evaluation index and data transmission evaluation index, the matching evaluation index of the allocated storage node is calculated using the following formula: In the formula, Indicates the matching evaluation index of the allocated storage node, , , are the weights of memory usage evaluation index, computing performance evaluation index, and data transmission evaluation index in the calculation of matching evaluation index, respectively, and , , The specific value of is determined by the hierarchical analysis method.

7. The distributed storage electronic data processing system according to claim 1, characterized in that: The process of using genetic algorithms to optimize the initial population is as follows: each allocation scheme in the initial population is regarded as an individual, and each individual is sorted from small to large according to the fitness value. The top 50% of individuals are selected as parents for crossover and mutation operations to generate new individuals. The new individuals and parents are used as the new population, and the fitness value of each individual in the new population is calculated. The selection, crossover and mutation operations are repeated until the optimization ends when the predetermined number of iterations is reached. The allocation scheme corresponding to the minimum fitness value after the optimization is completed is taken as the optimal allocation scheme.

Images