Distributed data management system and data storage method

Abstract

The present application discloses a distributed data management system and a data storage method. The present application obtains index column data of the result set data by dynamically sampling the result set data, and then obtains cost sample data of the result set data by sampling the index column data; a data volume estimation unit obtains a data volume score of the result set data by calculating the data volume estimation model based on the cost sample data and in combination with a data volume estimation model; a data warehousing processing unit stores the result set data in a storage node according to the data storage mode corresponding to the comparison result based on the comparison result of the data volume score and a preset data volume threshold value and the correspondence between the comparison result and the data storage mode. Different storage modes are adopted based on the different data volumes, thereby solving the technical problems of resource waste and low calculation efficiency in the processing of result sets in the existing distributed data management system.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a distributed data management system and a data storage method. Background Art

[0002] With the development of big data technology, finding the intrinsic connection between data through the correlation between data has become one of the most effective means to achieve business growth, making data analysis business favored by more and more enterprises and units.

[0003] In the process of data analysis, the data system has a large amount of data entry work every moment. When the association conditions cannot be clearly predicted, data of various structures are constantly associated and collided to generate new result sets. Different result sets are again continuously associated and collided to continue to generate new result sets.

[0004] Currently, most mainstream data systems are distributed data management systems, which distribute tasks as evenly as possible to all processing nodes for processing, thereby reducing the resource overhead of each processing node in processing data. However, existing distributed data management systems still have technical problems such as resource waste and low computing efficiency in processing result sets. Summary of the invention

[0005] The present application provides a distributed data management system and a data storage method, which are used to solve the technical problems that the existing distributed data management system still has resource waste and low computing efficiency in processing result sets.

[0006] The first aspect of the present application provides a distributed data management system, comprising:

[0007] A result set data acquisition unit, used for acquiring result set data;

[0008] A first preprocessing unit is used to dynamically sample the result set data to obtain index column data of the result set data, and then perform sampling processing on the index column data to obtain cost sample data of the result set data;

[0009] A data volume estimation unit, configured to obtain a data volume score of the result set data by calculating the data volume estimation model based on the cost sample data and in combination with the data volume estimation model, wherein the data volume estimation model is a mathematical model trained based on historical cost sample data samples;

[0010] A data storage processing unit is used to store the result set data into a storage node according to the data storage method corresponding to the comparison result of the data volume score and the preset data volume threshold, combined with the correspondence between the comparison result and the data storage method, wherein the data storage method includes: distributed storage method and single-point storage method.

[0011] Preferably, it also includes:

[0012] A mixed operation processing unit is used to redistribute or broadcast the second result set data in accordance with the distributed storage method when a mixed operation based on the first result set data and the second result set data occurs, wherein the first result set data is the result set data stored in the distributed storage method, and the second result set data is the result set data stored in the single point storage method.

[0013] Preferably, it also includes:

[0014] The second preprocessing unit is used to input the result set data into the storage node so that the result set data is processed by the storage node, and then the processed result set data is distributed from the storage node to the storage node, wherein the storage node is used to clean, escape and filter the result set data.

[0015] Preferably, the result set data specifically includes: one or more data tables, and one data table corresponds to one task.

[0016] Preferably, it also includes:

[0017] A resource request scheduling processing unit is used to respond to a received resource scheduling request, determine the instance process required for the execution of the resource scheduling request, call a number of storage units according to the instance process, so as to execute the instance process through the storage units, and complete the resource scheduling task corresponding to the resource scheduling request, wherein the number of the storage units is not less than the number of the instance processes, and each of the storage units executes only one instance process.

[0018] Preferably, the ratio of the instance process to the number of users corresponding to the instance process is 1:2.

[0019] Preferably, the storage nodes are communicatively connected in sequence to form a ring topology.

[0020] Preferably, it also includes:

[0021] The replica data management unit is used to store the replica data generated by the source storage node into the replica storage node, wherein the replica storage node is N storage nodes arranged along a preset direction of the source storage node, and the number of the replica storage nodes corresponds to the number of the replica data.

[0022] Preferably, it also includes:

[0023] A data recovery unit is used to determine adjacent storage nodes of the failed storage node based on the ring topology when data loss occurs in the storage node, and to recover data of the failed storage node according to the data of the adjacent storage nodes.

[0024] A second aspect of the present application provides a data storage method, which is applied to a distributed data management system as mentioned in the first aspect of the present application, comprising:

[0025] The result set data acquisition unit acquires the result set data;

[0026] A first preprocessing unit dynamically samples the result set data to obtain index column data of the result set data, and then performs sampling processing on the index column data to obtain cost sample data of the result set data;

[0027] The data volume estimation unit obtains the data volume score of the result set data by calculating the data volume estimation model based on the cost sample data and the data volume estimation model, wherein the data volume estimation model is a mathematical model trained based on historical cost sample data samples;

[0028] The data warehousing processing unit stores the result set data into a storage node according to the data storage method corresponding to the comparison result of the data volume score and the preset data volume threshold value, combined with the correspondence between the comparison result and the data storage method, wherein the data storage method includes: distributed storage method and single-point storage method.

[0029] It can be seen from the above technical solutions that this application has the following advantages:

[0030] This application adopts different storage methods based on the different data sizes. Single-point storage is used for data with small data volumes, which will only be saved in a certain node, while distributed storage is used for data with large data volumes, which will be sliced ​​into multiple nodes for storage. It avoids the need for distributed overhead for all computing operations. In the case of "independent computing benefits > distributed scheduling costs + distributed computing costs", independent computing can completely avoid distributed scheduling costs and waste system resources. When "distributed computing time efficiency > distributed computing costs > independent computing time", the system will implement distributed scheduling, which greatly improves the efficiency of computing and solves the technical problems of resource waste and low computing efficiency in the processing of result sets in existing distributed data management systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A schematic diagram of the structure of an embodiment of a distributed data management system provided by the present application;

[0033] Figure 2 A schematic diagram of a flow chart of an embodiment of a data storage method provided by the present application;

[0034] Figure 3 A schematic diagram of data storage effect of a distributed data management system and a data storage method based on the present application;

[0035] Figure 4 This is a schematic diagram of the resource request scheduling architecture of the distributed data management system of this application;

[0036] Figure 5 The number of socket connections measured based on the resource request scheduling processing method of this application;

[0037] Figure 6 The number of socket connections measured based on the existing resource request scheduling processing method;

[0038] Figure 7 This is a schematic diagram of the data table structure in the distributed data management system of this application. DETAILED DESCRIPTION

[0039] The essence of any distributed system is to "distribute a processing task as evenly as possible to all processing nodes" and "reduce the network overhead of data transmission between nodes as much as possible". However, due to the extreme skewness of the result set, for result sets at the GB or TB level, distributing the task to all nodes for parallel computing is undoubtedly an optimized solution. However, for data at the MB or KB level, the system overhead brought by data distribution is far greater than the system overhead of processing data (distributed overhead refers to the physical resources used by the system to complete this distributed scheduling). This phenomenon is the main reason for the technical problems of resource waste and low computing efficiency in the processing of result sets in existing distributed data management systems.

[0040] The embodiments of the present application provide a distributed data management system and a data storage method, which are used to solve the technical problems that the existing distributed data management system still has resource waste and low computing efficiency in processing result sets.

[0041] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] See also Figure 1 The first embodiment of the present application provides a distributed data management system, including:

[0043] A result set data acquisition unit 101, used to acquire result set data;

[0044] The first preprocessing unit 102 is used to dynamically sample the result set data to obtain index column data of the result set data, and then perform sampling processing on the index column data to obtain cost sample data of the result set data;

[0045] The data volume estimation unit 103 is used to obtain the data volume score of the result set data by calculating the data volume estimation model based on the cost sample data and the data volume estimation model, wherein the data volume estimation model is a mathematical model trained based on the historical cost sample data samples;

[0046] The data storage processing unit 104 is used to store the result set data into the storage node according to the data storage method corresponding to the comparison result of the data volume score and the preset data volume threshold, combined with the correspondence between the comparison result and the data storage method, wherein the data storage method includes: distributed storage method and single-point storage method.

[0047] In the distributed data management system provided in the present application, the storage method is different based on the different data sizes of the computing resources. When the data is entered, the program will automatically sample and analyze the generated index columns and save them as cost sample data. The program will estimate the data volume. If it exceeds a certain threshold, the data will be sliced ​​and distributed. If it is below a certain threshold, the data will not be sliced ​​and saved as a single node. At this time, the calculation of slice resources is distributed calculation, and the calculation of non-slice resources is single-point calculation. This design avoids the need for distributed overhead for all computing operations. In the case of "independent computing benefits > distributed scheduling costs + distributed computing costs", independent computing can completely avoid distributed scheduling costs and avoid waste of system resources. When "distributed computing time efficiency > distributed computing costs > independent computing time", the system will implement distributed scheduling, which greatly improves the efficiency of computing. Its storage effect can be found in Figure 3 .

[0048] The above is a detailed description of the first embodiment of a distributed data management system provided by the present application. The following is a detailed description of the second embodiment of a distributed data management system provided by the present application.

[0049] See also Figure 1 The second embodiment of the present application provides a distributed data management system based on the above first embodiment, including:

[0050] Furthermore, it also includes:

[0051] The mixed operation processing unit 105 is used to redistribute or broadcast the second result set data in a distributed storage manner when a mixed operation based on the first result set data and the second result set data occurs, wherein the first result set data is result set data stored in a distributed storage manner, and the second result set data is result set data stored in a single-point storage manner.

[0052] It should be noted that, based on the classified storage and classified operations provided by the first embodiment above, when complex mixed operations occur, for example, when the first result set data stored in a distributed storage manner is associated with the second result set data stored in a single-point storage manner, the program will broadcast or redistribute the second result set data internally, distribute independent operations into distributed operations, and improve the efficiency of executing mixed operations.

[0053] Furthermore, it also includes:

[0054] The second preprocessing unit 106 is used to input the result set data into the storage node, process the result set data through the storage node, and then distribute the processed result set data from the storage node to the storage node, wherein the storage node is used to clean, escape and filter the result set data.

[0055] It should be noted that the distributed data management system provided by the present application may also include: a second preprocessing unit, which is used to input the result set data into the storage node before storing the result set data into the storage node, and after the storage node performs further preprocessing, the result set data is distributed from the storage node to the storage node, thereby reducing the processing performance overhead of the storage node. For example, the preprocessing performed by the storage node on the result set data includes: (processing data that does not meet the storage requirements), escaping (performing data escape processing), and filtering (filtering data that does not meet the conditions).

[0056] In the scenario where data is entered into the warehouse one by one, the data entry node is also used for data distribution, and the performance overhead of the storage node is reduced by forwarding data without passing through the storage node.

[0057] Furthermore, the result set data specifically includes: one or more data tables, and one data table corresponds to one task.

[0058] It should be noted that if Figure 7 As shown, each data resource has an independent container (defined as a table in the database) for processing. The result set data of each task is saved in an independent table without the need for an index structure. The query analysis calculation is based on continuous IO, which can easily realize the function of migration, archiving and hierarchical storage. It can solve the problems of arbitrary fluctuations in the size of the created result set, arbitrary and random association rules between result sets, extreme skewness of result set data, unclear predicate queries, and unpredictable association optimization, which make the "index" design optimization mechanism in any database completely ineffective.

[0059] Furthermore, it also includes:

[0060] The resource request scheduling processing unit 107 is used to respond to the received resource scheduling request, determine the instance process required for the resource scheduling request to execute, and call a number of storage units according to the instance process to execute the instance process through the storage units to complete the resource scheduling task corresponding to the resource scheduling request, wherein the number of storage units is not less than the number of instance processes, and each storage unit executes only one instance process.

[0061] Furthermore, the ratio of instance processes to the number of users corresponding to the instance processes is 1:2.

[0062] It should be noted that if Figure 4As shown, the embodiment of the present application adopts a concurrent processing design of a single physical node and a single instance. One request will schedule the connection request resources of "number of physical nodes*2". One storage node only runs one instance process, and each instance process connects to multiple users. Compared with the traditional database running multiple instance processes on one storage node, the resource request concurrent processing mechanism of the present application can save more database socket connections and improve system resource utilization. For specific test results, please refer to Figure 5 and Figure 6 .

[0063] Among them, in the embodiment of the present application, each instance process is preferably connected to two users, and the instance process with more user connection requirements can be distributed to multiple storage nodes for execution, further improving the utilization rate of system resources.

[0064] Furthermore, each storage node is connected in communication in sequence to form a ring topology.

[0065] Furthermore, it also includes:

[0066] The replica data management unit 108 is used to store the replica data generated by the source storage node into the replica storage nodes, wherein the replica storage nodes are N storage nodes arranged along a preset direction of the source storage node, and the number of the replica storage nodes corresponds to the number of the replica data.

[0067] Furthermore, it also includes:

[0068] The data recovery unit 109 is used to determine the adjacent storage nodes of the failed storage node based on the ring topology structure when data loss occurs in the storage node, and recover the data of the failed storage node according to the data of the adjacent storage nodes.

[0069] It should be noted that the system provided in the embodiment of the present application is also provided with a replica data management mechanism, and its two adjacent physical nodes will form a partial replica relationship, and the replica structure of the entire system forms a ring, such as ABCA, for example: T1 data is in server A, and its replica T1' is in server B; T2 data is in server B, and its replica data T2' is in server C; T3 data is in server C, and its replica T3' is in server A. When the number of replicas is N, data loss will only occur if all N adjacent servers fail. When a single node fails, the system will not generate background operations for data redistribution, thereby avoiding the problem of system performance degradation caused by redistribution. When the failed node is back online, the recovery action is only performed between the failed node and its adjacent nodes, which can reduce the fluctuation of node failure on other nodes and the recovery action is more efficient.

[0070] The above is a detailed description of a first embodiment of a distributed data management system provided by the present application. The following is a detailed description of an embodiment of a data storage method provided by the present application and applied in the above-mentioned distributed data management system.

[0071] See also Figure 2 The third embodiment of the present application provides a data storage method, which is applied to a distributed data management system as mentioned in the first embodiment or the second embodiment of the present application, including:

[0072] Step 201: The result set data acquisition unit acquires result set data;

[0073] Step 202: The first preprocessing unit dynamically samples the result set data to obtain index column data of the result set data, and then performs sampling processing on the index column data to obtain cost sample data of the result set data;

[0074] Step 203: The data volume estimation unit obtains the data volume score of the result set data by calculating the data volume estimation model based on the cost sample data and the data volume estimation model, wherein the data volume estimation model is a mathematical model trained based on the historical cost sample data samples;

[0075] Step 204, the data storage processing unit stores the result set data into the storage node according to the data storage method corresponding to the comparison result of the data volume score and the preset data volume threshold value, combined with the correspondence between the comparison result and the data storage method, wherein the data storage method includes: distributed storage method and single-point storage method.

[0076] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0080] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed data management system, characterized in that: include: A result set data acquisition unit, used to acquire result set data, wherein the result set data is generated based on data of different structures or different existing result sets through mutual correlation and collision; A first preprocessing unit is used to dynamically sample the result set data to obtain index column data of the result set data, and then perform sampling processing on the index column data to obtain cost sample data of the result set data; A data volume estimation unit, configured to obtain a data volume score of the result set data by calculating the data volume estimation model based on the cost sample data and in combination with the data volume estimation model, wherein the data volume estimation model is a mathematical model trained based on historical cost sample data samples; The data storage processing unit is used to store the result set data in a storage node in a single-point storage manner if the data volume score is lower than the data volume threshold based on the comparison result between the data volume score and the preset data volume threshold; if the data volume score exceeds the data volume threshold, the result set data is stored in the storage node in a distributed storage manner.

2. A distributed data management system according to claim 1, characterized in that: Also includes: A mixed operation processing unit is used to redistribute or broadcast the second result set data in accordance with the distributed storage method when a mixed operation based on the first result set data and the second result set data occurs, wherein the first result set data is the result set data stored in the distributed storage method, and the second result set data is the result set data stored in the single point storage method.

3. A distributed data management system according to claim 1, characterized in that: Also includes: The second preprocessing unit is used to input the result set data into the storage node so that the result set data is processed by the storage node, and then the processed result set data is distributed from the storage node to the storage node, wherein the storage node is used to clean, escape and filter the result set data.

4. A distributed data management system according to claim 1, characterized in that: The result set data specifically includes: one or more data tables, and one data table corresponds to one task.

5. A distributed data management system according to claim 1, characterized in that: Also includes: A resource request scheduling processing unit is used to respond to a received resource scheduling request, determine the instance process required for the execution of the resource scheduling request, call a number of storage units according to the instance process, so as to execute the instance process through the storage units, and complete the resource scheduling task corresponding to the resource scheduling request, wherein the number of the storage units is not less than the number of the instance processes, and each of the storage units executes only one instance process.

6. A distributed data management system according to claim 5, characterized in that: The ratio of the instance process to the number of users corresponding to the instance process is 1:

2.

7. A distributed data management system according to claim 1, characterized in that: The storage nodes are connected in sequence to form a ring topology.

8. A distributed data management system according to claim 7, characterized in that: Also includes: The replica data management unit is used to store the replica data generated by the source storage node into the replica storage node, wherein the replica storage node is N storage nodes arranged along a preset direction of the source storage node, and the number of the replica storage nodes corresponds to the number of the replica data.

9. A distributed data management system according to claim 8, characterized in that: Also includes: A data recovery unit is used to determine adjacent storage nodes of the failed storage node based on the ring topology when data loss occurs in the storage node, and to recover data of the failed storage node according to the data of the adjacent storage nodes.

10. A data storage method, applied to a distributed data management system as claimed in any one of claims 1 to 9, characterized in that: include: Acquiring result set data, wherein the result set data is generated based on data of different structures or different existing result sets through mutual correlation and collision; Dynamically sampling the result set data to obtain index column data of the result set data, and then performing sampling processing on the index column data to obtain cost sample data of the result set data; According to the cost sample data, in combination with a data volume estimation model, a data volume score of the result set data is obtained by operation of the data volume estimation model, wherein the data volume estimation model is a mathematical model trained based on historical cost sample data samples; According to the comparison result between the data volume score and the preset data volume threshold, if the data volume score is lower than the data volume threshold, the result set data is stored in the storage node in a single-point storage manner; if the data volume score exceeds the data volume threshold, the result set data is stored in the storage node in a distributed storage manner.

Images