A method and device for slicing
Through logarithmic algorithms, the sharding and segmentation of HBase clusters has been solved, and the low resource utilization and hot issues in the existing technology have been achieved, achieving more efficient resource utilization and performance improvement.
Patent Information
- Application Number
- CN201810134829.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-02-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2038-02-09
AI Technical Summary
The existing HBase data sharding management method cannot effectively utilize cluster resources, and there are hot issues and small file problems, resulting in low read and write efficiency and cluster performance.
The logarithmic algorithm is used for sharding and segmentation management. By obtaining the maximum shard size of the current target table on the server, and calculating the shard size threshold based on the number of shards, we decide whether to perform sharding to avoid hotspots and small file problems.
It effectively improves the resource utilization rate of the HBase cluster, solves data hot issues, significantly improves read and write performance, and reduces cluster load.
Smart Images

Figure CN110134326B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a slicing method and device. Background Art
[0002] HBase (HBase is a highly reliable, high-performance, column-oriented, scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap server.) technology has been widely used in the field of big data. Its characteristics are that it can horizontally expand the scale of the cluster and evenly distribute a large amount of data on each server in the cluster, thereby achieving very high read and write performance. However, the existing HBase data sharding management method cannot effectively utilize cluster resources and has hot spot problems.
[0003] There are two common methods for HBase sharding:
[0004] The first method is the fixed value algorithm, which uses a fixed shard size. For example, each shard is set to 50G. When the size of any data shard reaches 50G, the HBase system will split the shard into two 25G shards. This method is simple and easy to implement, but when the amount of data is less than 50G, hot spots will occur. This shard is stored on a server in the cluster, and only this server is busy in the cluster, while other servers are idle. The resource utilization of the cluster is very low, and the advantages of HBase distribution cannot be brought into play.
[0005] The second method is the cubic algorithm, and the formula is: basic shard size * 2 * number of shards ^ 3. First, set a basic shard size, such as 128M. Then check the number of shards of the specified table on the current server, for example, there is 1 shard; substitute the formula: 128M*2*1^3=256M. The meaning is to split the shard when the shard size reaches 256M, and split it into two shards of 128M in size. By analogy, the shard size is 128M when there is 1 shard, 1024M when there are 2 shards, and 3456M when there are 3 shards. Until the shard size reaches the maximum value we set, for example, the maximum shard is 40G. This method solves the hot spot problem of the first method mentioned above, but it leads to new problems. It will generate many small files below 10G, which will seriously affect the performance of the HBase cluster.
[0006] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:
[0007] There are two main problems with the existing HBase sharding methods: the first fixed value algorithm will produce hot spots, that is, when the data volume is small, the data will be concentrated on a data shard on a server, which will result in low read and write efficiency and uneven cluster resource utilization. The second cubic algorithm will produce small file problems. According to the cubic algorithm, many small data shards will be generated, which are essentially many small files. The efficiency of reading and writing queries on these small files is very low, affecting the performance of the cluster. Summary of the invention
[0008] In view of this, an embodiment of the present invention provides a sharding method and device, which can effectively improve cluster resource utilization and solve the data hotspot problem.
[0009] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a sharding method is provided, comprising obtaining the size of the largest shard of the current target table on the server; determining that the size of the largest shard is greater than a shard size threshold, and then executing sharding of the shard; wherein the shard size threshold is obtained according to the number of shards of the current target table on the server.
[0010] Optionally, the shard size threshold is obtained by a logarithmic algorithm according to the number of shards of the current target table on the server.
[0011] Optionally, according to the number of shards of the current target table on the server, the shard size threshold is obtained by a logarithmic algorithm, including:
[0012] Check the number of shards of the current target table on the server;
[0013] Calculate the shard size threshold: shard size threshold = ln (number of shards) * disk space + shard base; where the disk space and shard base are obtained through the configuration file on the current server.
[0014] Optionally, the method further includes: when the size of the largest shard is less than or equal to a shard size threshold, waiting for a preset time period and then obtaining the size of the largest shard of the current target table on the server.
[0015] Optionally, the server is a server in an HBase cluster.
[0016] In addition, according to one aspect of an embodiment of the present invention, a sharding device is provided, including an acquisition module for acquiring the size of the largest shard of the current target table on the server; a sharding module for determining that the size of the largest shard is greater than a shard size threshold, and then executing sharding of the shard; wherein the shard size threshold is obtained according to the number of shards of the current target table on the server.
[0017] Optionally, the segmentation module obtains the shard size threshold by a logarithmic algorithm according to the number of shards of the current target table on the server.
[0018] Optionally, the slicing module obtains the slicing size threshold by a logarithmic algorithm according to the number of slicings of the current target table on the server, including:
[0019] Check the number of shards of the current target table on the server;
[0020] Calculate the shard size threshold: shard size threshold = ln (number of shards) * disk space + shard base; where the disk space and shard base are obtained through the configuration file on the current server.
[0021] Optionally, the acquisition module is further configured to: when the size of the largest shard is less than or equal to a shard size threshold, wait for a preset time period and then acquire the size of the largest shard of the current target table on the server.
[0022] Optionally, the server is a server in an HBase cluster.
[0023] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including:
[0024] one or more processors;
[0025] a storage device for storing one or more programs,
[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above-mentioned sharding embodiments.
[0027] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any of the above-mentioned sharding embodiments is implemented.
[0028] One embodiment of the above invention has the following advantages or beneficial effects: the present invention uses a logarithmic algorithm to manage sharding, solving the hotspot problem and small file problem of HBase cluster sharding. Thus, the performance and cluster resource rate of the HBase cluster are improved, the client read and write performance is greatly improved, and the load of the HBase cluster is reduced.
[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0031] Figure 1 is a schematic diagram of the main process of the sharding method according to an embodiment of the present invention;
[0032] Figure 2 It is a schematic diagram of the main process of the sharding method according to the reference embodiment of the present invention;
[0033] Figure 3 is a comparison chart of the shard sizes (in MB) according to a reference embodiment of the present invention;
[0034] Figure 4 is a comparison chart of small slices (in GB) according to a reference embodiment of the present invention;
[0035] Figure 5 is a schematic diagram of main modules of a sharding device according to an embodiment of the present invention;
[0036] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0037] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0039] Figure 1 is a sharding method according to an embodiment of the present invention, such as Figure 1 As shown, the sharding method includes:
[0040] Step S101, obtaining the size of the largest shard of the current target table on the server.
[0041] Preferably, the server is a server in an HBase cluster.
[0042] Step S102: if it is determined that the size of the largest shard is greater than a shard size threshold, the shard is split. The shard size threshold is obtained according to the number of shards of the current target table on the server.
[0043] Preferably, the shard size threshold is obtained by a logarithmic algorithm according to the number of shards of the current target table on the server.
[0044] Preferably, when obtaining the shard size threshold according to the number of shards of the current target table on the server, the shard size threshold can be obtained by detecting the number of shards of the current target table on the server and then calculating according to the following formula.
[0045] Shard size threshold = ln(number of shards)*disk space+shard base.
[0046] Furthermore, the disk space and shard base in the formula are obtained through the configuration file on the current server.
[0047] It is also worth noting that when the size of the largest fragment is less than or equal to the fragment size threshold, you can wait for a preset time period and then execute step S101 and step S102.
[0048] According to the various embodiments above, it can be seen that the sharding method described above solves the hotspot problem of the HBase cluster by using a logarithmic algorithm to divide the shards of the HBase cluster, so that the HBase cluster resources are fully utilized. At the same time, it can solve the hotspot problem of the shards and greatly improve the read and write performance of the cluster.
[0049] Figure 2 1 is a schematic diagram of the main process of a sharding method according to a reference embodiment of the present invention, which is applied to a server. The sharding method may include:
[0050] Step S201, obtain the configuration file of the HBase cluster stored on the current server to read the disk space and shard base.
[0051] The sharding base is usually set to 128M or 256M. The initial sharding base is usually relatively small. When the amount of data is small, the shards can be split as quickly as possible so that the shards are evenly distributed on each node in the cluster.
[0052] Disk space: The unit is GB. It is the disk space that the HBase cluster needs to occupy on the server for data storage. It is usually equal to the total capacity of the disk on a server.
[0053] Step S202, detecting the number of shards of the current target table on the server.
[0054] The number of shards refers to the number of shards of the target table on the current server.
[0055] Step S203, calculating the slice size threshold.
[0056] In an embodiment, the shard size threshold for slicing the current server is calculated based on the number of shards, disk space, and shard base, specifically:
[0057] Shard size threshold = ln(number of shards)*disk space+shard base.
[0058] Step S204, obtaining the size of the largest shard of the current target table on the server.
[0059] Step S205, determine whether the size of the largest fragment is greater than the fragment size threshold, if so, proceed to step S206, otherwise proceed to step S207.
[0060] Step S206, execute segmentation, and return to step S201.
[0061] Step S207, wait for a preset time period, and then return to step S201.
[0062] As an embodiment, after waiting for a preset period of time, step S201 to step S207 may be continued, that is, the shards of the target table may continue to be segmented.
[0063] It is worth noting that, according to the present invention, from the following fragment size comparison diagram (such as Figure 3 As shown in the figure, it can be seen that the straight line in the comparison chart is a constant value algorithm that will produce data hot spots; the broken line in the comparison chart is a cubic algorithm that will produce a large number of small fragments in the early stage of data expansion; and the curved line in the comparison chart is a logarithmic algorithm that solves the hot spot problem and the small fragment problem very well.
[0064] In addition, as can be seen from the following table, the slicing method adopted by the present invention is different from the fixed value algorithm and the legislative algorithm in the prior art.
[0065] Comparison table of shard sizes (in MB):
[0066]
[0067]
[0068] At the same time, from the following small slice comparison chart (such as Figure 4 As shown), it can be seen that:
[0069] The constant value algorithm has no small shards, but has serious hot spot problems; the cubic algorithm has 8% of shards smaller than 10G, which seriously affects the performance of the cluster; the logarithmic algorithm has only 2% of shards smaller than 10G, and as the amount of data increases, large shards always occupy a higher proportion.
[0070] The above small shards can also refer to the following table:
[0071] Small shards (in GB) comparison table:
[0072] Shard size Fixed value algorithm Cubic Algorithm Logarithmic Algorithm <10G 0% 8% 2% 10-20G 0% 2% 4% 20-30G 0% 2% 6% 30-40G 0% 0% 10% >=40G 100% 88% 78%
[0073] Figure 5 is a sharding device according to an embodiment of the present invention, such as Figure 5 As shown, the sharding device 500 includes an acquisition module 501 and a sharding module 502. The acquisition module 501 can acquire the size of the largest shard of the current target table on the server. Then the sharding module 502 determines that the size of the largest shard is greater than the sharding size threshold, and then performs sharding on the shard. The sharding size threshold is obtained according to the number of shards of the current target table on the server.
[0074] Furthermore, the server is a server in an HBase cluster.
[0075] As a preferred embodiment, the shard size threshold is obtained by a logarithmic algorithm according to the number of shards of the current target table on the server. Further, when obtaining the shard size threshold according to the number of shards of the current target table on the server, the shard size threshold can be obtained by detecting the number of shards of the current target table on the server and then calculating according to the following formula.
[0076] Shard size threshold = ln(number of shards)*disk space+shard base.
[0077] Preferably, the disk space and shard base in the formula are obtained through a configuration file on the current server.
[0078] It is also worth noting that when the size of the largest shard is less than or equal to the shard size threshold, the acquisition module 501 may wait for a preset period of time and then acquire the size of the largest shard of the current target table on the server.
[0079] It should be noted that the specific implementation content of the slicing device described in the present invention has been described in detail in the slicing method described above, so the repeated content will not be described here.
[0080] Figure 6 An exemplary system architecture 600 to which the method or apparatus for sharding according to an embodiment of the present invention can be applied is shown. Figure 6 An exemplary system architecture 600 is shown to which the sharding method or sharding apparatus according to an embodiment of the present invention may be applied.
[0081] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0082] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).
[0083] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.
[0084] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.
[0085] It should be noted that the sharding method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the sharding device is generally arranged in the server 605 .
[0086] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0087] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0088] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0089] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.
[0090] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.
[0091] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0092] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0093] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module and a segmentation module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.
[0094] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining the size of the largest shard of the current target table on the server; determining that the size of the largest shard is greater than a shard size threshold, then executing the segmentation of the shard; wherein the shard size threshold is obtained according to the number of shards of the current target table on the server.
[0095] The technical solution according to the embodiment of the present invention can effectively improve cluster resource utilization and solve the data hotspot problem.
[0096] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A slicing method, characterized in that: include: Get the size of the largest shard of the current target table on the server; If it is determined that the size of the largest shard is greater than the shard size threshold, the shard is split; wherein, according to the number of shards of the current target table on the server, the shard size threshold is obtained by a logarithmic algorithm; according to the number of shards of the current target table on the server, the shard size threshold is obtained by a logarithmic algorithm, including: detecting the number of shards of the current target table on the server; calculating the shard size threshold: shard size threshold = ln (number of shards) * disk space + shard base; wherein, the disk space and the shard base are obtained through the configuration file on the current server.
2. The method according to claim 1, characterized in that Also includes: When the size of the largest shard is less than or equal to the shard size threshold, wait for a preset period of time and then obtain the size of the largest shard of the current target table on the server.
3. The method according to any one of claims 1-2, characterized in that: The server is a server in the HBase cluster.
4. A slicing device, characterized in that: include: The acquisition module is used to obtain the size of the largest shard of the current target table on the server; A slicing module is used to determine that the size of the largest slicing is greater than a slicing size threshold, and then execute slicing of the slicing; wherein, the slicing module obtains the slicing size threshold by a logarithmic algorithm according to the number of slicings of the current target table on the server; the slicing module obtains the slicing size threshold by a logarithmic algorithm according to the number of slicings of the current target table on the server, including: detecting the number of slicings of the current target table on the server; calculating the slicing size threshold: slicing size threshold = ln (number of slicings) * disk space + slicing cardinality; wherein, the disk space and slicing cardinality are obtained through the configuration file on the current server.
5. The device according to claim 4, characterized in that The acquisition module is further used for: When the size of the largest shard is less than or equal to the shard size threshold, wait for a preset period of time and then obtain the size of the largest shard of the current target table on the server.
6. The device according to any one of claims 4-5, characterized in that: The server is a server in the HBase cluster.
7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.
8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.
Citation Information
Patent Citations
File uploading method and device
CN107172131A
File downloading method, device, server and terminal
CN107360254A
Distributed data processing method, device and system, and server
CN107508901A
Elasticsearch cluster expansion method supporting resource leveling
CN107566531A
System and methods for utilization-based balancing of traffic to an information retrieval system
US9641431B1
