HASH-BUCKET-based big data node capacity expansion and contraction processing method and system

Abstract

The invention discloses a big data node capacity expansion and contraction processing method and system based on HASH-BUCKET, and the method comprises the following steps: S1, reading metadata information stored in a control node metadatabase, and obtaining metadata node information; s2, judging whether node resource updating succeeds or not, if yes, entering the step S3, and if not, entering the step S1; s3, calculating spatial information of a target node based on a HASH-BUCKET algorithm, judging whether a metadata node is updated or not, if yes, entering the step S4, and otherwise, entering the step S1; and S4, creating a DELTA temporary area, migrating and updating metadata node information, updating a Hash key value mapping relation, writing into a storage area, and recovering and cleaning the DELTA temporary area.

Description

Technical Field

[0001] The present invention relates to the technical field of big data storage, and more particularly to a method and system for processing capacity expansion and contraction of big data nodes based on HASH-BUCKET. Background Art

[0002] At present, most information data is stored in a distributed manner (localized) with distributed computing; horizontal expansion, through the expansion of cluster nodes, big data nodes add or reduce servers. When the original data nodes cannot meet the data storage needs, it is necessary to dynamically add nodes on the basis of the original cluster, that is, dynamically add servers. When adding servers, there is no need to restart the cluster, and there is no shared architecture at all. Expansion (reduction) and data redistribution are common operation and maintenance management operations, which are time-consuming and labor-intensive to manage. The expansion and reduction of large data warehouses often require manual maintenance, which cannot be completed dynamically, and it is impossible to accurately judge whether resources need to be released or increased. In the information age, how to achieve "light interaction" in data storage, make data storage more reasonable and effective, and conform to the "convenient" and "fast" labels of social development has become a research direction for technicians in this field. Summary of the invention

[0003] The purpose of the present invention is to provide a large data node expansion and contraction processing method based on HASH-BUCKET, in order to solve the problem that the cluster space cannot be dynamically expanded and contracted in the background technology. Due to the increasing data, the available space of hdfs becomes smaller, resulting in the inability to store subsequent data in hdfs, and the data directory of hdfs is closely related to the space size of the volume group. Therefore, the present invention adopts a large data node expansion and contraction processing method based on HASH-BUCKET, which will be divided into two scenarios to realize expansion and contraction, and is suitable for capacity changes when clusters are built.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for scaling up and down a big data node based on HASH-BUCKET includes the following steps:

[0006] S1: Read the metadata information stored in the control node metadata database to obtain metadata node information;

[0007] S2: Determine whether the node resource update is successful, if so, proceed to step S3, otherwise proceed to step S1;

[0008] S3: Calculate the spatial information of the target node based on the HASH-BUCKET algorithm and determine whether the metadata node has been updated. If yes, proceed to step S4; otherwise, proceed to step S1.

[0009] S4: Create a DELTA temporary area, migrate and update metadata node information, update the Hash key-value mapping relationship, write to the storage area, and recycle and clean up the DELTA temporary area.

[0010] In some embodiments, the S1: reading metadata information stored in the control node metadata database to obtain metadata node information; includes: before the expansion operation, the metadata information must be read first to obtain the workspace area of ​​the data node.

[0011] In some embodiments, the step S2: determining whether the node resource update is successful, if so, proceeding to step S3, otherwise proceeding to step S1; includes:

[0012] If the node needs to be updated, the size of the expansion space is compared with the space in the storage pool. If the expansion space is smaller than the space in the storage pool, the expansion request is accepted; if the expansion space is larger than the space in the storage pool, the expansion request is rejected.

[0013] In some embodiments, the S3: calculates the spatial information of the target node based on the HASH-BUCKET algorithm, including: writing the information of the data node into the delta temporary area, and the storage pool plans a matching location space according to the HASH-BUCKET algorithm based on the space request of the delta temporary area.

[0014] In some embodiments, the S4: creates a DELTA temporary area, migrates and updates metadata node information, updates the Hash key value mapping relationship, writes to the storage area, and recycles and cleans up the DELTA temporary area; includes: expansion and reduction; the expansion includes:

[0015] If the matching location space planned by the HASH-BUCKET algorithm is occupied, it will be managed in the hash bucket through a linked list. Specifically, when multiple nodes are mapped to the same bucket, they will be inserted into the linked list in sequence, and the small metadata node will be selected for migration, excluding the data storage directory specified by the corresponding node configuration file. The storage pool will write the mount information of the selected data space into the delta temporary area. The data node will migrate the working node path according to the information in the delta temporary area, and update the mapping relationship, so that the data node can be expanded.

[0016] In some embodiments, the shrinking includes: metadata reading out the mapping position, adding a delta temporary area, adding a retired node to the delta position, metadata changing the mapping, and continuing the business process; Delta increases the length of the hash bucket according to the HASH-BUCKET algorithm, plans the hash value of the idle interval, and puts the node position mounted on the delta into the idle area to complete the entire shrinking process.

[0017] In some embodiments, the metadata node information includes the space size and storage address information required for metadata node storage.

[0018] In some embodiments, the control node is deployed separately or mixed with the data node.

[0019] In some embodiments, the DELTA temporary area is a hash table that records the difference values ​​of data changes, and only records the data from the last change to the current change.

[0020] This embodiment also provides a HASH-BUCKET-based big data node expansion and contraction processing system, which is used to implement any of the methods described above, including:

[0021] The data acquisition module is used to read the metadata information stored in the control node metadata database and obtain the metadata node information;

[0022] The data judgment module is used to judge whether the node resource update is successful;

[0023] The data calculation module calculates the spatial information of the target node based on the HASH-BUCKET algorithm and determines whether the metadata node has been updated based on the calculation results;

[0024] The data migration module is used to create a DELTA temporary area, migrate and update metadata node information, update the Hash key-value mapping relationship, write to the storage area, and recycle and clean up the DELTA temporary area.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention is a HASH-BUCKET-based big data node expansion and contraction processing method. Through the dynamic adjustment of existing partitions and management space, the data space can be used more efficiently, and the service interruption caused by mounting space is solved. The space is used dynamically and reasonably. When the original data node cannot meet the data storage demand, it is necessary to dynamically add nodes on the basis of the original cluster, that is, dynamically add servers. When adding servers, there is no need to restart the cluster. The efficiency and user experience are greatly improved. It is a new management mode that conforms to the current fast development rhythm of the times. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart for implementing dynamic expansion and contraction for HASH-BUCKET. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below in conjunction with the drawings in the preferred embodiments of the present application. In the drawings, the same or similar reference numerals throughout represent the same or similar parts or parts with the same or similar functions. The described embodiments are part of the embodiments of the present application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limitations on the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0029] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, or it can be an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0031] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "back", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are orientations or positional relationships based on the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0032] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or display that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed but may include other steps or elements not explicitly listed or inherent to such process, method, product, or display.

[0033] The following will be combined Figure 1 , a HASH-BUCKET-based big data node expansion and contraction processing method involved in the embodiment of the present application is described in detail. It is worth noting that the following embodiments are only used to explain the present application and do not constitute a limitation on the present application.

[0034] Embodiment 1:

[0035] See also Figure 1A method for scaling up and down a big data node based on HASH-BUCKET includes the following steps:

[0036] S1: Before starting the expansion and contraction planning, read the metadata information stored in the control node metadata database. The control node can be deployed separately or mixed with the data node to obtain the metadata node information; the metadata node information includes the space size and storage address information required for the metadata node storage.

[0037] S2: Determine whether the node resource update is successful. If so, proceed to step S3, otherwise proceed to S1; if the node needs to be updated, compare the size of the expansion space and the space in the storage pool. If the expansion space is smaller than the space in the storage pool, accept the expansion request. If the expansion space is larger than the space in the storage pool, reject the expansion request.

[0038] S3: Calculate the spatial information of the target node based on the HASH-BUCKET algorithm, determine whether the metadata node has been updated, if yes, proceed to step S4, otherwise proceed to S1; Calculate the spatial information of the target node based on the HASH-BUCKET algorithm, including: writing the information of the data node into the delta temporary area, and the storage pool plans the matching location space according to the HASH-BUCKET algorithm based on the space request of the delta temporary area. The algorithm model usually uses the hash value to calculate the number of buckets modulo.

[0039] S4: Create a DELTA temporary area. DELTA is a hash table that mainly records the difference value of data changes. To save space, only the data from the last change to the current change is recorded. Metadata node information is migrated and updated, the Hash key-value mapping relationship is updated, the storage area is written, and the DELTA temporary area is recycled and cleaned up; including: expansion and reduction; the expansion includes:

[0040] If the matching location space planned by the HASH-BUCKET algorithm is occupied, it will be managed in the hash bucket through a linked list. Specifically, when multiple nodes are mapped to the same bucket, they will be inserted into the linked list in sequence, and small metadata nodes will be migrated preferentially, and large metadata nodes will be avoided as much as possible. It should be noted that the data storage directory specified by the corresponding node configuration file is not included. The storage pool writes the mount information of the selected data space into the delta temporary area. The data node migrates the working node path according to the information in the delta temporary area and updates the mapping relationship, so that the data node can be expanded.

[0041] The HASH algorithm returns the bucket location based on the index key. The advantage is that no sorting is required in advance, and the search speed is not strongly related to the amount of data. Further compression can be done later based on space usage. At the same time, the setting of the delta temporary area is increased, and the mapping of the data node work configuration file is modified to facilitate subsequent one-click start and stop. At the same time, it can also ensure that normal business will not be affected by capacity changes.

[0042] After the expansion migration is completed, the delta temporary area is recovered to facilitate the implementation of the next expansion request. The storage nodes in this embodiment will only participate in the change when the expansion is completed. Its distribution position only changes the internal mapping of the storage node, and other data locations do not change, so it will not affect the business process. After the expansion is completed, the expansion node is removed from the storage pool, completing the entire expansion process.

[0043] The scaling-down process is as follows: metadata reads the mapping position, adds a delta temporary area, adds a retired node to the delta position, changes the metadata mapping, and continues the business process; Delta increases the length of the hash bucket according to the HASH-BUCKET algorithm, plans the hash value of the idle interval, puts the node position mounted on the delta into the idle area, deletes the delta area data, and adds the delta temporary area setting, mainly to cancel the binding with the business as soon as possible without affecting the operation of the business process, and finally releases the corresponding data space to complete the entire scaling-down process.

[0044] The invention adopts the HASH-BUCKET-based data node expansion and contraction, which is a processing method for dynamically processing available space. In view of the increasing data, the available space of hdfs becomes smaller, resulting in the inability to store subsequent data in hdfs. The invention will implement expansion and contraction in two scenarios, which is suitable for expansion and contraction when cluster is built. Note that the cluster should be in a closed state. Implementation scenario 1: Add partitions to the volume group to achieve expansion. After adding the hard disk, df and vgdisplay (linux system query command) do not check the added available space. Create partitions and physical volumes to add to the volume group where the hdfs data directory is located and allocate free space; Implementation scenario 2: The free space of the volume group where a certain file space is located is too large, so the volume group space is compressed for expansion.

[0045] Scenario 1 implements node expansion. When the data flow is too large, the storage address and storage capacity of the flow data are calculated through the HASH bucket, and the big data node is mounted to the corresponding data space to increase the capacity. When the current data flow does not need the address space, the corresponding reserved space is released, and the node links stored in the hash bucket are recycled so that other data flows can calculate and mount when they need space.

[0046] Scenario 2: To expand and reduce the disk space itself, if the free space of a file space is too large, the space can be managed through HASH-BUCKET, and the reserved space can be mounted to other file spaces in a logical way to achieve dynamic management of HASH-BUCKET. This will not only effectively manage the data space, but also fundamentally solve the problems of insufficient space and space redundancy.

[0047] The method of dynamically expanding and shrinking space through HASH-BUCKET management is a very efficient way to utilize space. It does not require adding servers and restarting the cluster, which will cause service interruptions. It allows users to fully utilize efficiency and utilization in the existing way.

[0048] This embodiment also provides a HASH-BUCKET-based big data node expansion and contraction processing system, which is used to implement any of the methods described above, including:

[0049] The data acquisition module is used to read the metadata information stored in the control node metadata database and obtain the metadata node information;

[0050] The data judgment module is used to judge whether the node resource update is successful;

[0051] The data calculation module calculates the spatial information of the target node based on the HASH-BUCKET algorithm and determines whether the metadata node has been updated based on the calculation results;

[0052] The data migration module is used to create a DELTA temporary area, migrate and update metadata node information, update the Hash key-value mapping relationship, write to the storage area, and recycle and clean up the DELTA temporary area.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for scaling up and down a big data node based on HASH-BUCKET, characterized in that: The following steps are involved: S1: Read the metadata information stored in the control node metadata database to obtain metadata node information; S2: Determine whether the node resource update is successful, if so, proceed to step S3, otherwise proceed to step S1; S3: Calculate the spatial information of the target node based on the HASH-BUCKET algorithm and determine whether the metadata node has been updated. If yes, proceed to step S4; otherwise, proceed to step S1. S4: Create a DELTA temporary area, migrate and update metadata node information, update the Hash key-value mapping relationship, write to the storage area, and recycle and clean up the DELTA temporary area.

2. According to a HASH-BUCKET-based large data node expansion and contraction processing method according to claim 1, it is characterized in that: The S1: reading metadata information stored in the control node metadata database to obtain metadata node information; including: before the expansion operation, the metadata information must be read first to obtain the workspace area of ​​the data node.

3. According to a HASH-BUCKET-based large data node expansion and contraction processing method according to claim 1, it is characterized in that: S2: Determine whether the node resource update is successful, if yes, proceed to step S3, otherwise proceed to step S1; including: If the node needs to be updated, the size of the expansion space is compared with the space in the storage pool. If the expansion space is smaller than the space in the storage pool, the expansion request is accepted; if the expansion space is larger than the space in the storage pool, the expansion request is rejected.

4. According to a HASH-BUCKET-based large data node expansion and contraction processing method according to claim 1, it is characterized in that: S3: Calculating the spatial information of the target node based on the HASH-BUCKET algorithm, including: writing the information of the data node into the delta temporary area, and the storage pool plans a matching location space according to the HASH-BUCKET algorithm based on the space request of the delta temporary area.

5. According to a HASH-BUCKET-based large data node expansion and contraction processing method according to claim 1, it is characterized in that: S4: creating a DELTA temporary area, migrating and updating metadata node information, updating the Hash key value mapping relationship, writing to the storage area, and reclaiming and cleaning the DELTA temporary area; including: expansion and reduction; the expansion includes: If the matching location space planned by the HASH-BUCKET algorithm is occupied, it will be managed in the hash bucket through a linked list. Specifically, when multiple nodes are mapped to the same bucket, they will be inserted into the linked list in sequence, and the small metadata node will be selected for migration, excluding the data storage directory specified by the corresponding node configuration file. The storage pool will write the mount information of the selected data space into the delta temporary area. The data node will migrate the working node path according to the information in the delta temporary area, and update the mapping relationship, so that the data node can be expanded.

6. According to a HASH-BUCKET-based large data node expansion and contraction processing method according to claim 1, it is characterized in that: The shrinking includes: metadata reading mapping position, adding delta temporary area, adding retired nodes to the delta position, metadata changing mapping, and continuing the business process; Delta increases the length of the hash bucket according to the HASH-BUCKET algorithm, plans the hash value of the idle interval, and puts the node position mounted on the delta into the idle area to complete the entire shrinking process.

7. A method for processing capacity expansion and contraction of a large data node based on HASH-BUCKET according to claim 1, characterized in that: The metadata node information includes the space size and storage address information required for metadata node storage.

8. The method for scaling up and down a large data node based on HASH-BUCKET according to claim 1, characterized in that: The control node is deployed independently or mixed with the data node.

9. The method for scaling up and down a large data node based on HASH-BUCKET according to claim 1, characterized in that: The DELTA temporary area is a hash table that records the difference value of data changes, and only records the data from the last change to the current change.

10. A big data node expansion and contraction processing system based on HASH-BUCKET, characterized in that: The method for implementing any one of claims 1 to 9 comprises: The data acquisition module is used to read the metadata information stored in the control node metadata database and obtain the metadata node information; The data judgment module is used to judge whether the node resource update is successful; The data calculation module calculates the spatial information of the target node based on the HASH-BUCKET algorithm and determines whether the metadata node has been updated based on the calculation results; The data migration module is used to create a DELTA temporary area, migrate and update metadata node information, update the Hash key-value mapping relationship, write to the storage area, and recycle and clean up the DELTA temporary area.

Images