Distributed database management method based on big data

Abstract

The invention belongs to the field of distributed databases, and discloses a distributed database management method based on big data, and the method comprises the steps: obtaining data blocks stored on a plurality of storage nodes of a distributed database based on a query request of a user; performing equivalent sorting on the data blocks in any two storage nodes to obtain a first data block range set and a second data block range set; constructing a pruning set based on the first data block range set; performing data block pruning on the first data block range set and the second data block range set based on the pruning set to obtain a first data block pruning set and a second data block pruning set; and performing data merging on the first data block range set and the second data block range set based on the first data block pruning set and the second data block pruning set to obtain merged data. According to the method, the connection cost between the two data sets and the additional network overhead in a distributed environment are greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the field of distributed databases, and in particular relates to a distributed database management method based on big data. Background Art

[0002] A distributed database is a database system in which data is distributed across multiple geographically located servers that are connected over a network to share and access data. Compared with traditional centralized databases, distributed database systems can provide higher availability, better performance, and scalability, and are therefore widely used in various industries.

[0003] Distributed database management involves multiple aspects such as data distribution management, transaction management, permission management, query optimization management, backup and recovery management, monitoring and maintenance management, etc.; among them, query optimization management is mainly used to optimize query execution plans and reduce cross-node communication and response time.

[0004] Since the original data in the distributed database is stored in shards, and the sort-merge join is the join operation algorithm between the relational data in the distributed database, during the query process, when the two data sets are large and not suitable for loading into the memory, the two records with equal join keys are merged through the sort-merge join to form part of the join result. The sort-merge join is an important option for the database query optimizer when selecting a join algorithm, especially when processing large data sets and optimizing disk I / O operations.

[0005] The existing sort-merge join method has at least the following defects: some data blocks in the original data do not need to be joined, which will increase additional overhead during the join operation, such as network overhead. In addition, when there is a large amount of data that needs to be joined, the probability of this part of data appearing increases, and the additional overhead increases accordingly. The existing technology cannot effectively process this part of the data blocks. Summary of the invention

[0006] The purpose of the present invention is to provide a distributed database management method based on big data, so as to solve the problem that the existing technology cannot effectively process these data blocks.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a distributed database management method based on big data, the method comprising: Obtaining data blocks stored on several storage nodes of a distributed database based on a user's query request; Sort the data blocks in any two storage nodes equally respectively to obtain a first data block range set and a second data block range set; Constructing a pruning set based on the first data block range set; Based on the pruning set, data blocks are pruned on the first data block range set and the second data block range set respectively to obtain a first data block pruned set and a second data block pruned set; The first data block range set and the second data block range set are merged based on the first data block pruning set and the second data block pruning set to obtain merged data.

[0008] Preferably, constructing a pruning set based on the first data block range set includes: Constructing a superset according to the minimum value and the maximum value of the data blocks in the first data block range set, wherein the first data block range set is a proper subset of the superset; Divide the superset into multiple sub-intervals based on a preset partitioning rule; Construct a pruning set based on multiple subintervals.

[0009] Preferably, the preset division rule is: Extracting the first data block quantity and the first data amount value in the first data block range set; Extracting the number of second data blocks and the second data volume value in the second data block range set; Determining a first data distribution degree based on the first data block quantity and the first data amount value; Determining a second data distribution degree based on the number of second data blocks and the second data amount value; Based on the first data distribution degree and the second data distribution degree, a segmentation quantity value is determined, where the segmentation quantity value is used to represent the number of times the superset is segmented.

[0010] Preferably, the calculation expression of the first data distribution degree is: ; In the formula, is the first data distribution degree, is the number of the first data block, is the first data value.

[0011] Preferably, the calculation expression of the second data distribution degree is: ; In the formula, is the second data distribution degree, is the number of the second data block, is the second data value.

[0012] Preferably, the expression of the segmentation quantity value is: ; In the formula, Dis the number of cuts, Indicates rounding up.

[0013] Preferably, data block pruning is performed on the first data block range set and the second data block range set respectively based on the pruning set to obtain the first data block pruning set and the second data block pruning set, including: Mapping a first data block in the first data block range set to the pruned set; Select any second data block from the second data block range set, determine whether the second data block matches the pruned set, and if not, discard the second data block; if so, determine whether the second data block matches the first data block mapped to the pruned set, and if not, discard the second data block; if so, retain the second data block, and add the second data block and the first data block matching the second data block to the screening set; Extracting a first data block in the screening set as a first data block pruning set; A second data block in the screening set is extracted as a second data block pruning set.

[0014] Preferably, merging the first data block range set and the second data block range set based on the first data block pruning set and the second data block pruning set to obtain merged data includes: Migrating each second data block in the second data block pruning set to a storage location corresponding to a first data block matching each second data block; At the storage positions corresponding to the first data blocks that match the second data blocks, locally merge the migrated second data blocks and the first data blocks that match the second data blocks to obtain locally merged data of each storage node; The local merged data of all storage nodes are globally merged to obtain the global merged data, and the global merged data is used as the merged data.

[0015] Preferably, the method further comprises: sending the combined data to a user.

[0016] Preferably, the method further comprises: Obtain original big data, and perform sharding processing on the original big data to obtain several data blocks; Several data blocks are stored on several storage nodes of the distributed database respectively.

[0017] In a second aspect, the present invention provides a distributed database management system based on big data, the system is used to implement the above-mentioned distributed database management method based on big data, and the system includes: A data acquisition module, used to acquire data blocks stored on several storage nodes of a distributed database based on a user's query request; A data sorting module, used to sort the data blocks in any two storage nodes for equal values, to obtain a first data block range set and a second data block range set; A pruning construction module, used to construct a pruning set based on the first data block range set; A data pruning module, configured to perform data block pruning on the first data block range set and the second data block range set respectively based on the pruning set to obtain a first data block pruning set and a second data block pruning set; The data merging module is used to merge the first data block range set and the second data block range set based on the first data block pruning set and the second data block pruning set to obtain merged data.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned distributed database management method based on big data when executing the computer program.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned distributed database management method based on big data.

[0020] Beneficial effects: After sorting the data blocks, the present invention constructs a pruning set, and uses the pruning set to prune the first data block range set and the second data block range set, so as to prune useless data blocks in the first data block range set and the second data block range set, which can greatly reduce the connection cost between the two data sets that need to be connected and merged, and reduce the additional network overhead in a distributed environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 It is a flow chart of a distributed database management method based on big data provided by one embodiment of the present invention; Figure 2 It is a block diagram of a distributed database management system based on big data provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0023] Embodiment 1 Figure 1 is a flow chart of a distributed database management method based on big data provided by an embodiment of the present invention, such as Figure 1 As shown, this embodiment provides a distributed database management method based on big data, and the method includes: Step S10: obtaining data blocks stored on several storage nodes of the distributed database based on the user's query request.

[0024] In this embodiment, the user generates a query request on the client, and the client uploads the query request to the server. The server parses the query request to clarify the data type and condition that the user wants to retrieve. After the query is parsed, the server needs to determine the storage node where the data block is located (a storage node has multiple storage locations, and each storage location has a data block). The query is executed in parallel on multiple nodes, and the locally stored data blocks are retrieved on each node. The queried data blocks are collected from each node, and all the data blocks are merged and sent to the user. Since some data blocks do not need to be merged during the merging process, the existence of these data blocks will increase the connection cost between the data blocks. Therefore, in order to reduce the connection cost between the two data blocks that need to be connected and merged, and reduce the additional network overhead in a distributed environment, the two data blocks are merged in the manner of steps S20 to S50.

[0025] Step S20: respectively sorting the data blocks in any two storage nodes for equal values ​​to obtain a first data block range set and a second data block range set.

[0026] In this embodiment, for two storage nodes that need to be connected, all data blocks on the two storage nodes are sorted according to equal sizes to obtain two sets, namely a first data block range set and a second data block range set; for example: for A and B that have a connection relationship, all data blocks on the storage node where A is located are sorted, and the value range of each data block is used as an element in the data block range set; at this time, an element in the first data block range set is the value range of a data block of A, and an element in the second data block range set is the value range of a data block of B.

[0027] Step S30: constructing a pruning set based on the first data block range set.

[0028] For step S30, specifically, constructing a pruning set based on the first data block range set includes: Step S310: constructing a superset according to the minimum value and the maximum value of the data blocks in the first data block range set, wherein the first data block range set is a proper subset of the superset; wherein the superset is a value range constructed with the minimum value of the data blocks in the first data block range set as the minimum value and the maximum value of the data blocks in the first data block range set as the maximum value, and each element in the first data block range set is in the superset; Step S320: dividing the superset into multiple sub-intervals based on a preset division rule, where the multiple sub-intervals are continuous value ranges and each sub-interval is evenly divided; Step S330: construct a pruning set according to the multiple sub-intervals; the multiple sub-intervals are combined together to obtain a pruning set, which is used to guide the pruning operation.

[0029] In step S320, as a further optimization of this embodiment, the preset division rule is: Step b10: Extract the number of first data blocks and the first data quantity value in the first data block range set, wherein the first data quantity value is calculated as follows: count the data quantity corresponding to the numerical range of each first data block in the first data block range set, where the data quantity refers to the number of rows in which the numerical range is arranged; sum the data quantities corresponding to all the first data block value ranges in the first data block range set to obtain the first data quantity value.

[0030] Step b20: Extract the number of second data blocks and the second data quantity value in the second data block range set, wherein the second data quantity value is calculated in the same way as the first data quantity value in step b10, i.e., the data quantity corresponding to the numerical range of each second data block in the second data block range set is counted; the data quantity corresponding to all the second data block value ranges in the second data block range set is summed to obtain the second data quantity value.

[0031] Step b30: Determine the first data distribution degree based on the number of first data blocks and the first data amount value; wherein the calculation expression of the first data distribution degree is: ; In the formula, is the first data distribution degree, is the number of the first data block, is the first data value.

[0032] Step b40: Determine the second data distribution degree based on the number of second data blocks and the second data amount value; wherein the calculation expression of the second data distribution degree is: ; In the formula, is the second data distribution degree, is the number of the second data block, is the second data value.

[0033] Step b50: Based on the first data distribution degree and the second data distribution degree, determine the number of splits, where the number of splits is used to represent the number of times the superset is split; wherein the expression of the number of splits is: ; In the formula, D is the number of cuts, Indicates rounding up; in this embodiment, the split quantity value is mainly used to control the granularity after the superset is split. The larger the split quantity value is, the more times the superset is split, the more sub-intervals are formed after the superset is split, and the smaller the granularity of the sub-intervals is.

[0034] In this embodiment, when the number of data blocks in the first data block range set or the second data block range set is large, but the amount of data in each data block is small, this indicates that there is uneven data distribution among the data blocks in the first data block range set or the second data block range set. At this time, the calculated first data distribution degree or second data distribution degree is relatively large, and thus the value of the number of splits is large. The larger the value of the number of splits, the more times the superset is split, the more sub-intervals are formed after the superset is split, and the smaller the granularity of the sub-intervals. The load associated with the position with unbalanced load can be distributed to other positions, thereby better solving the problem of data skew.

[0035] For example, when data is unevenly distributed, some nodes are overloaded. If the granularity of the sub-interval is larger, a large amount of data may be concentrated in a few sub-intervals, causing the nodes corresponding to these sub-intervals to be overloaded, while other nodes are underloaded. By selecting a larger number of splits, data can be more evenly distributed to each sub-interval, thereby alleviating the data skew problem and achieving load balancing.

[0036] Step S40: performing data block pruning on the first data block range set and the second data block range set respectively based on the pruning set to obtain a first data block pruning set and a second data block pruning set.

[0037] As a further optimization of this embodiment, data block pruning is performed on the first data block range set and the second data block range set respectively based on the pruning set to obtain the first data block pruning set and the second data block pruning set, including: Step S410: Map the first data block in the first data block range set to the pruned set; in this embodiment, perform an intersection operation on the numerical range of each first data block in the first data block range set and the sub-interval in the pruned set. If the intersection is not empty, it means that the first data block belongs to the current pruned set, and the first data block is associated with the corresponding sub-interval in the pruned set.

[0038] Step S420: select any second data block from the second data block range set, determine whether the second data block matches the pruned set, and if not, discard the second data block; if so, determine whether the second data block matches the first data block mapped to the pruned set, and if not, discard the second data block; if so, retain the second data block, and add the second data block and the first data block matching the second data block to the screening set; wherein, compare any second data block with the subinterval in the pruned set, determine whether the value range of the second data block and the subinterval in the pruned set are in a one-to-one relationship or a one-to-many relationship, and when it is a one-to-one relationship, add the second data block directly to the screening set. If it is one-to-many, it means that the value range of a second data block may correspond to multiple sub-intervals in the pruned set, and the value range of the second data block needs to be split to match the multiple sub-intervals in the pruned set. Since there are useless second data blocks in the second data block range set, the useless second data blocks will not be connected with the first data block in the first data block range set. In the splitting process, there are some second data blocks that have no intersection with the sub-intervals in the pruned set. This part of the second data block is useless and can be discarded, which can reduce the amount of data for subsequent local connection operations, reduce the calculation and storage costs, and reduce the network transmission cost. When the value range of the split second data block is within the corresponding sub-interval, the split second data block is mapped to the corresponding sub-interval.

[0039] After completing the mapping of the second data block, in each sub-interval, determine whether there is a first data block that intersects with the second data block. If so, it means that the second data block matches the first data block mapped to the pruned set, and the second data block is retained; if not, it means that the second data block does not match the first data block mapped to the pruned set, and the second data block needs to be discarded.

[0040] Step S430: extracting the first data block in the screening set as the first data block pruning set.

[0041] Step S440: extract the second data block in the screening set as the second data block pruning set.

[0042] Therefore, in the process of constructing the filter set, the pruning operation of useless data blocks is implemented, which can effectively reduce network overhead and improve the efficiency of subsequent data connection.

[0043] Step S50: merging the first data block range set and the second data block range set based on the first data block pruning set and the second data block pruning set to obtain merged data.

[0044] As a further optimization of this embodiment, in step S50, the first data block range set and the second data block range set are merged based on the first data block pruning set and the second data block pruning set to obtain merged data, including: Step S510: Migrate each second data block in the second data block pruning set to a storage location corresponding to a first data block matching each second data block; Step S520: locally merge each migrated second data block and the first data block matching each second data block at the storage location corresponding to the first data block matching each second data block, to obtain local merged data of each storage node; at this time, on one storage node, complete the local data merge of the data blocks at all storage locations on the storage node, to obtain local merged data of each storage node; Step S530: globally merge the local merged data of all storage nodes to obtain global merged data, and use the global merged data as merged data; in this embodiment, a node for global data merge operation can be established, and the local merged data obtained by layout merging on all storage nodes can be obtained on the node, and all the local merged data are globally merged to obtain final merged data. The final merged data is used as the user's query result, and the final merged data can be sent to the user.

[0045] As a further optimization of this embodiment, the method also includes: sending the merged data to the user. In this embodiment, the application scenarios of the method mainly include: a server and at least one client. A corresponding distributed database is built on the server, and steps S10 to S50 are executed on the server; the client can be a smart terminal such as a mobile phone, a PC, or a laptop computer. After obtaining the merged data, the server transmits the merged data to the client via a wireless network or a wired network, and the user can view the query results, i.e., the merged data, on the client.

[0046] As a further optimization of this embodiment, the method further includes: Step a10: obtaining original big data, and performing slicing processing on the original big data to obtain a number of data blocks; Step a20: Store the data blocks on the storage nodes of the distributed database respectively.

[0047] In this embodiment, whenever new original big data is generated, the original big data can be generated on the client and uploaded to the server by the client. The server segments the original big data to obtain several data blocks, and the server stores the data blocks on the corresponding storage nodes of the distributed database.

[0048] After sorting the data blocks, the present invention constructs a pruning set, and uses the pruning set to prune the first data block range set and the second data block range set, so as to prune useless data blocks in the first data block range set and the second data block range set, which can greatly reduce the connection cost between the two data sets that need to be connected and merged, and reduce the additional network overhead in a distributed environment.

[0049] Embodiment 2 Figure 2 is a block diagram of a distributed database management system based on big data provided by an embodiment of the present invention, such as Figure 2 As shown, this embodiment provides a distributed database management system based on big data, the system is used to implement the distributed database management method based on big data of embodiment 1, and the system includes: A data acquisition module, used to acquire data blocks stored on several storage nodes of a distributed database based on a user's query request; A data sorting module, used to sort the data blocks in any two storage nodes for equal values, to obtain a first data block range set and a second data block range set; A pruning construction module, used to construct a pruning set based on the first data block range set; A data pruning module, configured to perform data block pruning on the first data block range set and the second data block range set respectively based on the pruning set to obtain a first data block pruning set and a second data block pruning set; The data merging module is used to merge the first data block range set and the second data block range set based on the first data block pruning set and the second data block pruning set to obtain merged data.

[0050] After sorting the data blocks, the present invention constructs a pruning set, and uses the pruning set to prune the first data block range set and the second data block range set, so as to prune useless data blocks in the first data block range set and the second data block range set, which can greatly reduce the connection cost between the two data sets that need to be connected and merged, and reduce the additional network overhead in a distributed environment.

[0051] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the big data-based distributed database management method of the first embodiment when executing the computer program.

[0052] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the distributed database management method based on big data of the first embodiment is implemented.

[0053] After sorting the data blocks, the present invention constructs a pruning set, and uses the pruning set to prune the first data block range set and the second data block range set, so as to prune useless data blocks in the first data block range set and the second data block range set, which can greatly reduce the connection cost between the two data sets that need to be connected and merged, and reduce the additional network overhead in a distributed environment.

[0054] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0055] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A distributed database management method based on big data, characterized in that: The method comprises: Obtaining data blocks stored on several storage nodes of a distributed database based on a user's query request; Sort the data blocks in any two storage nodes equally respectively to obtain a first data block range set and a second data block range set; Constructing a pruning set based on the first data block range set; Based on the pruning set, data blocks are pruned on the first data block range set and the second data block range set respectively to obtain a first data block pruned set and a second data block pruned set; The first data block range set and the second data block range set are merged based on the first data block pruning set and the second data block pruning set to obtain merged data.

2. The distributed database management method based on big data according to claim 1 is characterized in that: Constructing a pruning set based on the first data block range set includes: Constructing a superset according to the minimum value and the maximum value of the data blocks in the first data block range set, wherein the first data block range set is a proper subset of the superset; Divide the superset into multiple sub-intervals based on a preset partitioning rule; Construct a pruning set based on multiple subintervals.

3. The distributed database management method based on big data according to claim 2 is characterized in that: The preset division rule is: Extracting the first data block quantity and the first data amount value in the first data block range set; Extracting the number of second data blocks and the second data volume value in the second data block range set; Determining a first data distribution degree based on the first data block quantity and the first data amount value; Determining a second data distribution degree based on the number of second data blocks and the second data amount value; Based on the first data distribution degree and the second data distribution degree, a segmentation quantity value is determined, where the segmentation quantity value is used to represent the number of times the superset is segmented.

4. The distributed database management method based on big data according to claim 3 is characterized in that: The calculation expression of the first data distribution degree is: ； In the formula, is the first data distribution degree, is the number of the first data block, is the first data value.

5. The distributed database management method based on big data according to claim 4 is characterized in that: The calculation expression of the second data distribution degree is: ； In the formula, is the second data distribution degree, is the number of the second data block, is the second data value.

6. The distributed database management method based on big data according to claim 5 is characterized in that: The expression of the segmentation quantity value is: ； In the formula, D is the number of cuts, Indicates rounding up.

7. The distributed database management method based on big data according to claim 2 is characterized in that: Based on the pruning set, data blocks are pruned on the first data block range set and the second data block range set respectively to obtain the first data block pruned set and the second data block pruned set, including: Mapping a first data block in the first data block range set to the pruned set; Select any second data block from the second data block range set, determine whether the second data block matches the pruned set, and if not, discard the second data block; if so, determine whether the second data block matches the first data block mapped to the pruned set, and if not, discard the second data block; if so, retain the second data block, and add the second data block and the first data block matching the second data block to the screening set; Extracting a first data block in the screening set as a first data block pruning set; A second data block in the screening set is extracted as a second data block pruning set.

8. The distributed database management method based on big data according to claim 7 is characterized in that: Merging the first data block range set and the second data block range set based on the first data block pruning set and the second data block pruning set to obtain merged data includes: Migrating each second data block in the second data block pruning set to a storage location corresponding to a first data block matching each second data block; At the storage positions corresponding to the first data blocks that match the second data blocks, locally merge the migrated second data blocks and the first data blocks that match the second data blocks to obtain locally merged data of each storage node; The local merged data of all storage nodes are globally merged to obtain the global merged data, and the global merged data is used as the merged data.

9. The distributed database management method based on big data according to claim 1, characterized in that: The method further includes sending the merged data to a user.

10. The distributed database management method based on big data according to claim 1, characterized in that: The method further comprises: Obtain original big data, and perform sharding processing on the original big data to obtain several data blocks; Several data blocks are stored on several storage nodes of the distributed database respectively.

Images