Regional data distribution method applied to open source database and related equipment

Abstract

The embodiment of the invention belongs to the technical field of data processing, and relates to a regional data distribution method and related equipment applied to an open source database, and the method comprises the steps: reading partition data of a target data warehouse tool according to a big data analysis engine, and obtaining an elastic distributed data set; performing segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniform segmentation boundary data; performing re-partitioning operation on the elastic distributed data set according to the uniform cutting boundary data to obtain re-partitioning data; performing data mapping operation on the re-partitioned data to obtain an hfile; creating a pre-partition table in the target open source database according to the uniform cutting boundary data; and importing the hfile into the pre-partition table. According to the method and the device, the Major Compaction operation which consumes much resources in the operation of the hbase cluster and the query hotspot problem caused by non-uniform distribution of region data are effectively avoided.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a regional data allocation method and related equipment applied to an open source database. Background Art

[0002] A common solution for deriving massive amounts of data from Hive to HBase is to use Spark to import Hive data directly into HBase, and then the data service accesses HBase to provide higher performance data query speed.

[0003] However, the applicant has found that the above derivative function often causes uneven data distribution in the region of hbase, which easily leads to major compaction operations that are very resource-intensive. In addition, because data is unevenly distributed to each region in hbase, it causes data query hot spots. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a regional data allocation method and related equipment applied to an open source database to solve the problem of data query hotspots in traditional derivative functions.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a regional data allocation method applied to an open source database, which adopts the following technical solution:

[0006] The partition data of the target data warehouse tool is read by the big data analysis engine to obtain a resilient distributed data set;

[0007] Performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly segmented boundary data;

[0008] Repartitioning the elastic distributed data set according to the uniform cutting boundary data to obtain repartitioned data;

[0009] Performing data mapping operation on the repartitioned data to obtain an hfile file;

[0010] Creating a pre-partitioned table in the target open source database according to the uniformly cut boundary data;

[0011] Import the hfile file into the pre-partition table.

[0012] Furthermore, the step of performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly segmented boundary data specifically includes the following steps:

[0013] Determine the initial data volume for each region based on the performance of the target open source database;

[0014] Performing an optimization operation on each of the regions according to the total amount of derivatives of the target data warehouse tool and the initial data amount, to obtain the number of optimal regions and the optimal data amount corresponding to each of the optimal regions;

[0015] Extracting primary keys from the elastic distributed data set and sorting the primary keys to obtain KeyRdd;

[0016] Get the total number of all partitions;

[0017] Calculate the uniform data distribution quantity according to the total quantity of all partitions and the optimal data quantity corresponding to each optimal area;

[0018] The elastic distributed data set is evenly divided according to the uniform data allocation quantity to obtain the uniform cutting boundary data.

[0019] Furthermore, the step of performing a data mapping operation on the repartitioned data to obtain an hfile file specifically includes the following steps:

[0020] The data mapping operation is distributed to several nodes for parallel execution according to a distributed computing framework.

[0021] Furthermore, the step of importing the hfile file into the pre-partition table specifically includes the following steps:

[0022] Perform a merge operation on the hfile file to obtain a merged file;

[0023] Import the merged file into the pre-partition table.

[0024] Furthermore, it is characterized in that after the step of reading the partition data of the target data warehouse tool according to the big data analysis engine to obtain the elastic distributed data set, and before the step of performing a segmentation operation on the elastic distributed data set according to the segmentation algorithm to obtain the uniformly cut boundary data, the following steps are also included:

[0025] Perform data cleaning, data conversion and data filtering operations on the elastic distributed data set.

[0026] Furthermore, the step of importing the hfile file into the pre-partition table specifically includes the following steps:

[0027] Calling the established data quality monitoring mechanism to monitor the real-time data quality indicators of the import operation in real time;

[0028] If the real-time data quality indicator is abnormal, an alarm mechanism and reverse data repair and reprocessing operations are triggered.

[0029] In order to solve the above technical problems, the embodiment of the present application further provides a regional data allocation device applied to an open source database, which adopts the following technical solution:

[0030] A data reading module is used to read the partitioned data of the target data warehouse tool according to the big data analysis engine to obtain a resilient distributed data set;

[0031] A segmentation module, used for performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly cut boundary data;

[0032] A repartitioning module, used for performing a repartitioning operation on the elastic distributed data set according to the uniform cutting boundary data to obtain repartitioning data;

[0033] A data mapping module is used to perform a data mapping operation on the repartitioned data to obtain an hfile file;

[0034] A pre-partition table creation module, used to create a pre-partition table in a target open source database according to the uniformly cut boundary data;

[0035] The file import module is used to import the hfile file into the pre-partition table.

[0036] Furthermore, the segmentation module includes:

[0037] An initial data volume determination submodule, used to determine the initial data volume of each region according to the performance of the target open source database;

[0038] An optimization submodule, configured to perform an optimization operation on each of the regions according to the total amount of derivatives of the target data warehouse tool and the initial data amount, so as to obtain the number of optimal regions and the optimal data amount corresponding to each of the optimal regions;

[0039] A primary key sorting submodule, used for extracting primary keys from the elastic distributed data set and sorting the primary keys to obtain KeyRdd;

[0040] The submodule for obtaining the total number of partitions is used to obtain the total number of all partitions;

[0041] A distribution quantity calculation submodule, used for calculating the uniform data distribution quantity according to the total quantity of all partitions and the optimal data quantity corresponding to each optimal area;

[0042] The even-dividing submodule is used to evenly divide the elastic distributed data set according to the even data distribution quantity to obtain the even-cut boundary data.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0044] The system comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the regional data allocation method applied to an open source database as described above when executing the computer-readable instructions.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0046] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the regional data allocation method applied to an open source database as described above are implemented.

[0047] The present application provides a regional data allocation method applied to an open source database, including: reading the partition data of a target data warehouse tool according to a big data analysis engine to obtain an elastic distributed data set; performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly cut boundary data; performing a repartitioning operation on the elastic distributed data set according to the uniformly cut boundary data to obtain repartitioned data; performing a data mapping operation on the repartitioned data to obtain an hfile file; creating a pre-partition table in a target open source database according to the uniformly cut boundary data; and importing the hfile file into the pre-partition table. Compared with the prior art, the present application evenly segments the data in hive, and obtains a segmentation boundary array according to a segmentation algorithm, and then establishes a pre-partition table based on the boundary array, generates a corresponding hfile file from the evenly segmented data, and bulk loads it into hbase, so that the data of the derivative hbase table in multiple hbase regions is evenly distributed, effectively avoiding the MajorCompaction operation that consumes a lot of resources during the operation of the hbase cluster and the query hotspot problem caused by uneven distribution of region data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0050] Figure 2is a flowchart of an implementation method of a regional data allocation method applied to an open source database provided in an embodiment of the present application;

[0051] Figure 3 is a structural diagram of a regional data distribution device applied to an open source database provided in an embodiment of the present application;

[0052] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0059] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0060] It should be noted that the regional data allocation method applied to the open source database provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the regional data allocation device applied to the open source database is generally set in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0062] Continue to refer Figure 2 , shows a flow chart of an embodiment of a regional data allocation method applied to an open source database according to the present application. The regional data allocation method applied to an open source database comprises: step S201, step S202, step S203, step S204, step S205 and step S206.

[0063] In step S201, the partition data of the target data warehouse tool is read according to the big data analysis engine to obtain a resilient distributed data set.

[0064] In an embodiment of the present application, hive (data warehouse tool) partition data is read through spark (big data analysis engine) to obtain dataRdd (resilient distributed data set).

[0065] In step S202, a segmentation operation is performed on the elastic distributed data set according to a segmentation algorithm to obtain uniformly cut boundary data.

[0066] In step S203, the elastic distributed data set is repartitioned according to the uniformly cut boundary data to obtain repartitioned data.

[0067] In an embodiment of the present application, dataRdd (resilient distributed data set) is combined with a uniformly cut boundary data array for repartitioning.

[0068] In step S204, a data mapping operation is performed on the repartitioned data to obtain an hfile file.

[0069] In an embodiment of the present application, the data in the repartition is mapped into a data format suitable for storage in hbase (open source database) and saved as hfile.

[0070] In step S205, a pre-partition table is created in the target open source database according to the uniformly cut boundary data.

[0071] In the embodiment of the present application, hbase (open source database) pre-partitioning and table creation are performed according to the cutting boundary array array.

[0072] In step S206, the hfile file is imported into the pre-partition table.

[0073] In an embodiment of the present application, the generated hfile files of uniform size are bulkloaded into a pre-partition table.

[0074] In an embodiment of the present application, a regional data allocation method applied to an open source database is provided, including: reading the partition data of a target data warehouse tool according to a big data analysis engine to obtain an elastic distributed data set; performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly cut boundary data; performing a repartitioning operation on the elastic distributed data set according to the uniformly cut boundary data to obtain repartitioned data; performing a data mapping operation on the repartitioned data to obtain an hfile file; creating a pre-partition table in a target open source database according to the uniformly cut boundary data; and importing the hfile file into the pre-partition table. Compared with the prior art, the present application evenly segments the data in hive, and obtains a segmentation boundary array according to a segmentation algorithm, and then establishes a pre-partition table based on the boundary array, and generates a corresponding hfile file from the evenly segmented data and bulk loads it into hbase, so that the data of the derivative hbase table in multiple hbase regions is evenly distributed, effectively avoiding the Major Compaction operation that consumes a lot of resources during the operation of the hbase cluster and the query hotspot problem caused by uneven distribution of region data.

[0075] In some optional implementations of the embodiments of the present application, the above-mentioned step of performing a segmentation operation on the elastic distributed data set according to the segmentation algorithm to obtain uniformly cut boundary data specifically includes the following steps:

[0076] Determine the initial data volume for each region based on the performance of the target open source database;

[0077] Performing optimization operations on each region according to the total amount of derivatives of the target data warehouse tool and the amount of initial data, obtaining the number of optimal regions and the optimal amount of data corresponding to each optimal region;

[0078] Extract the primary key from the elastic distributed dataset and sort the primary key to obtain KeyRdd;

[0079] Get the total number of all partitions;

[0080] Calculate the uniform data distribution amount according to the total number of all partitions and the optimal data amount corresponding to each optimal area;

[0081] The elastic distributed data set is evenly divided according to the uniform data allocation quantity to obtain uniform cutting boundary data.

[0082] In the embodiment of the present application, the data size of each region is determined according to the performance of hbase (open source database), generally 1 to 2G, and the number of regions region-quantity is obtained by dividing the total size of the derivative in hive (data warehouse tool) by the optimal data volume of a single region.

[0083] In an embodiment of the present application, the primary key key is extracted from dataRdd (elastic distributed data set), sorted according to the key, and keyRdd is obtained.

[0084] In the embodiment of the present application, the partition index and the number of data in each partition, as well as the total number of data in all partitions data-total-count are obtained.

[0085] In the embodiment of the present application, the number of data uniformly distributed in each region is obtained: region-count = data-total-count / region-quantity.

[0086] In the embodiment of the present application, keyRdd is evenly divided into region-count parts to obtain an array of evenly cut boundary data.

[0087] In some optional implementations of the embodiments of the present application, the above step of performing a data mapping operation on the repartitioned data to obtain the hfile file specifically includes the following steps:

[0088] According to the distributed computing framework, data mapping operations are distributed to several nodes for parallel execution.

[0089] In the embodiments of the present application, the distributed computing framework is mainly used to divide large computing tasks into multiple small parts and execute these tasks in parallel on multiple computers (or nodes) in a network. This framework usually provides a set of tools and interfaces for task scheduling, data distribution, communication between nodes, and collection and integration of results. Common distributed computing frameworks include Apache Hadoop, Apache Spark, Apache Flink, etc.

[0090] In the embodiments of the present application, a data mapping operation refers to a process of converting data from one form to another. In the context of distributed computing, a data mapping operation may involve filtering, converting, aggregating, or otherwise processing data. As an example, for example, a large data set needs to be split into multiple small blocks, and the same conversion operation is performed on each small block.

[0091] In the embodiment of the present application, in the distributed computing framework, the original data set is divided into multiple parts, each of which is assigned to one or more nodes for processing. This process is usually automatically managed by the framework, which assigns tasks based on the availability and processing power of the nodes.

[0092] In an embodiment of the present application, when performing data mapping operations on repartitioned data, a distributed computing framework, such as MapReduce or Spark, is used to distribute data mapping tasks to multiple nodes for parallel execution. By increasing computing resources, the data mapping process is accelerated, the mapping time is shortened, and the real-time performance of data processing is improved.

[0093] In some optional implementations of the embodiments of the present application, the above step of importing the hfile file into the pre-partition table specifically includes the following steps:

[0094] Perform a merge operation on the hfile file to obtain a merged file;

[0095] Import the merged file into the pre-partitioned table.

[0096] In the embodiment of the present application, HFile merging is the process of merging multiple small HFile files into a large HFile file. The purpose of doing so is to reduce the disk space occupied and improve data access performance. By merging, the number of files can be reduced, the complexity of addressing when reading data can be reduced, and thus the reading efficiency can be improved.

[0097] In the embodiment of the present application, the merging operation may be RatioBasedCompactionPolicy, ExploringCompactionPolicy, FIFOCompactionPolicy, DateTieredCompactionPolicy, etc. Each policy has its specific triggering conditions and merging rules.

[0098] In the embodiment of the present application, when importing hfile files into the pre-partition table, a batch import method is adopted to merge multiple hfile files into a larger file before importing, thereby reducing the number of imports and improving the import efficiency. At the same time, the parallel writing capability of the distributed file system is utilized to realize the parallel import of hfile files, further speeding up the import speed.

[0099] In some optional implementations of the embodiments of the present application, after the step of reading the partition data of the target data warehouse tool according to the big data analysis engine to obtain the elastic distributed data set, and before the step of performing a segmentation operation on the elastic distributed data set according to the segmentation algorithm to obtain the uniformly cut boundary data, the following steps are also included:

[0100] Perform data cleaning, data conversion, and data filtering operations on elastic distributed data sets.

[0101] In an embodiment of the present application, in order to optimize data mapping and import performance, a preprocessing mechanism is introduced to perform preprocessing operations such as cleaning, conversion and filtering on the original data before the data enters the data warehouse, thereby removing redundant and invalid data, reducing the amount of data, narrowing the scope of data mapping and import, and improving the efficiency of the entire data processing process.

[0102] In some optional implementations of the embodiments of the present application, the above step of importing the hfile file into the pre-partition table specifically includes the following steps:

[0103] Call the established data quality monitoring mechanism to monitor the real-time data quality indicators of the import operation in real time;

[0104] If there are abnormalities in the real-time data quality indicators, the alarm mechanism and reverse data repair and reprocessing operations will be triggered.

[0105] In the embodiments of the present application, building a data quality monitoring mechanism generally includes the following aspects:

[0106] Define data quality indicators: Based on business needs and data characteristics, define a series of indicators for measuring data quality, such as data completeness, accuracy, consistency, timeliness, uniqueness, etc.

[0107] Develop monitoring tools: Use programming languages ​​(such as Java, Python, etc.) and database technologies (such as SQL, NoSQL, etc.) to develop tools or platforms for real-time monitoring of data quality indicators.

[0108] Set monitoring rules: Set corresponding monitoring rules based on data quality indicators, such as triggering an alarm when the data missing rate exceeds a certain proportion, and performing deduplication processing when the data duplication rate exceeds a threshold.

[0109] Integration and deployment: Integrate monitoring tools into the data processing process and deploy them on the corresponding computing resources to achieve real-time monitoring and alarm functions.

[0110] In the embodiments of the present application, real-time monitoring refers to the process of collecting, analyzing and displaying data quality indicators in real time through monitoring tools, including the following steps:

[0111] Data collection: Use database queries, API calls, and other methods to collect key data in the data processing process in real time.

[0112] Data analysis: Analyze the collected data in real time, calculate the value of data quality indicators, and compare them with the preset monitoring rules.

[0113] Result display: The analysis results are displayed to users in real time in the form of charts, logs, etc., so that users can understand the data quality status in a timely manner.

[0114] In an embodiment of the present application, a data quality monitoring and alarm mechanism is established to monitor data quality indicators in the data processing process in real time, such as data integrity and consistency. If data quality problems are found, an alarm is triggered, and data repair and reprocessing measures are taken in time to ensure the accuracy and reliability of data processing, thereby providing a high-quality data foundation for subsequent data analysis.

[0115] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0116] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0118] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0119] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a regional data allocation device applied to an open source database. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0120] like Figure 3 As shown, the regional data distribution device 200 applied to the open source database in the embodiment of the present application includes:

[0121] The data reading module 210 is used to read the partition data of the target data warehouse tool according to the big data analysis engine to obtain a resilient distributed data set;

[0122] A segmentation module 220 is used to perform segmentation operations on the elastic distributed data set according to a segmentation algorithm to obtain uniformly cut boundary data;

[0123] A repartitioning module 230 is used to perform a repartitioning operation on the elastic distributed data set according to the uniform cutting boundary data to obtain repartitioned data;

[0124] The data mapping module 240 is used to perform data mapping operation on the repartitioned data to obtain an hfile file;

[0125] A pre-partition table creation module 250 is used to create a pre-partition table in the target open source database according to the uniformly cut boundary data;

[0126] The file import module 260 is used to import the hfile file into the pre-partition table.

[0127] In an embodiment of the present application, a regional data distribution device 200 applied to an open source database is provided, including: a data reading module 210, used to read the partition data of the target data warehouse tool according to the big data analysis engine to obtain a resilient distributed data set; a segmentation module 220, used to perform a segmentation operation on the resilient distributed data set according to the segmentation algorithm to obtain uniform cutting boundary data; a repartitioning module 230, used to perform a repartitioning operation on the resilient distributed data set according to the uniform cutting boundary data to obtain repartitioned data; a data mapping module 240, used to perform a data mapping operation on the repartitioned data to obtain an hfile file; a pre-partition table creation module 250, used to create a pre-partition table in the target open source database according to the uniform cutting boundary data; a file import module 260, used to import the hfile file into the pre-partition table. Compared with the prior art, the present application evenly divides the data in hive, obtains the dividing boundary array according to the dividing algorithm, then establishes a pre-partition table based on the boundary array, generates the corresponding hfile file from the evenly divided data and bulk loads it into hbase, so that the data of the derived hbase table in multiple hbase regions is evenly distributed, effectively avoiding the Major Compaction operation that consumes a lot of resources during the operation of the hbase cluster and the query hot spot problem caused by the uneven distribution of region data.

[0128] In some optional implementations of the embodiments of the present application, the segmentation module includes:

[0129] An initial data volume determination submodule is used to determine the initial data volume of each region according to the performance of the target open source database;

[0130] An optimization submodule, used for performing optimization operation on each region according to the total amount of derivatives of the target data warehouse tool and the amount of initial data, to obtain the number of optimal regions and the optimal amount of data corresponding to each optimal region;

[0131] The primary key sorting submodule is used to extract the primary key from the elastic distributed data set and sort the primary key to obtain KeyRdd;

[0132] The submodule for obtaining the total number of partitions is used to obtain the total number of all partitions;

[0133] The allocation quantity calculation submodule is used to calculate the uniform data allocation quantity according to the total quantity of all partitions and the optimal data quantity corresponding to each optimal area;

[0134] The equalization submodule is used to perform an equalization operation on the elastic distributed data set according to the uniform data distribution quantity to obtain uniform cutting boundary data.

[0135] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0136] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0137] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0138] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for the regional data allocation method applied to the open source database, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0139] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as running the computer-readable instructions of the regional data allocation method applied to the open source database.

[0140] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0141] The computer device provided by the present application evenly divides the data in hive, obtains the dividing boundary array according to the dividing algorithm, then establishes a pre-partition table based on the boundary array, generates corresponding hfile files for the evenly divided data, and bulk loads them into hbase, so that the data of the derived hbase table in multiple hbase regions is evenly distributed, effectively avoiding the Major Compaction operation that consumes a lot of resources during the operation of the hbase cluster and the query hot spot problem caused by the uneven distribution of region data.

[0142] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the regional data allocation method applied to an open source database as described above.

[0143] The computer-readable storage medium provided by the present application evenly divides the data in hive, obtains the dividing boundary array according to the dividing algorithm, then establishes a pre-partition table based on the boundary array, generates corresponding hfile files for the evenly divided data, and bulk loads them into hbase, so that the data of the derived hbase table in multiple hbase regions is evenly distributed, effectively avoiding the Major Compaction operation that consumes a lot of resources during the operation of the hbase cluster and the query hot spot problem caused by the uneven distribution of region data.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0145] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A regional data allocation method applied to an open source database, characterized in that: The steps include: The partition data of the target data warehouse tool is read by the big data analysis engine to obtain a resilient distributed data set; Performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly segmented boundary data; Repartitioning the elastic distributed data set according to the uniform cutting boundary data to obtain repartitioned data; Performing data mapping operation on the repartitioned data to obtain an hfile file; Creating a pre-partitioned table in the target open source database according to the uniformly cut boundary data; Import the hfile file into the pre-partition table.

2. The regional data allocation method applied to an open source database according to claim 1, characterized in that: The step of performing a segmentation operation on the elastic distributed data set according to the segmentation algorithm to obtain uniformly segmented boundary data specifically includes the following steps: Determine the initial data volume for each region based on the performance of the target open source database; Performing an optimization operation on each of the regions according to the total amount of derivatives of the target data warehouse tool and the initial data amount, to obtain the number of optimal regions and the optimal data amount corresponding to each of the optimal regions; Extracting primary keys from the elastic distributed data set and sorting the primary keys to obtain KeyRdd; Get the total number of all partitions; Calculate the uniform data distribution quantity according to the total quantity of all partitions and the optimal data quantity corresponding to each optimal area; The elastic distributed data set is evenly divided according to the uniform data allocation quantity to obtain the uniform cutting boundary data.

3. The regional data allocation method applied to an open source database according to claim 1, characterized in that: The step of performing a data mapping operation on the repartitioned data to obtain the hfile file specifically includes the following steps: The data mapping operation is distributed to several nodes for parallel execution according to a distributed computing framework.

4. The regional data allocation method applied to an open source database according to claim 1, characterized in that: The step of importing the hfile file into the pre-partition table specifically includes the following steps: Perform a merge operation on the hfile file to obtain a merged file; Import the merged file into the pre-partition table.

5. The regional data allocation method applied to an open source database according to claim 1, characterized in that: After the step of reading the partition data of the target data warehouse tool according to the big data analysis engine to obtain the elastic distributed data set, and before the step of performing a segmentation operation on the elastic distributed data set according to the segmentation algorithm to obtain uniformly cut boundary data, the following steps are also included: Perform data cleaning, data conversion and data filtering operations on the elastic distributed data set.

6. The regional data allocation method applied to an open source database according to claim 1, characterized in that: The step of importing the hfile file into the pre-partition table specifically includes the following steps: Calling the established data quality monitoring mechanism to monitor the real-time data quality indicators of the import operation in real time; If the real-time data quality indicator is abnormal, an alarm mechanism and reverse data repair and reprocessing operations are triggered.

7. A regional data distribution device applied to an open source database, characterized in that: include: A data reading module is used to read the partitioned data of the target data warehouse tool according to the big data analysis engine to obtain a resilient distributed data set; A segmentation module, used for performing a segmentation operation on the elastic distributed data set according to a segmentation algorithm to obtain uniformly cut boundary data; A repartitioning module, used for performing a repartitioning operation on the elastic distributed data set according to the uniform cutting boundary data to obtain repartitioning data; A data mapping module is used to perform a data mapping operation on the repartitioned data to obtain an hfile file; A pre-partition table creation module, used to create a pre-partition table in a target open source database according to the uniformly cut boundary data; The file import module is used to import the hfile file into the pre-partition table.

8. The regional data distribution device applied to an open source database according to claim 7, characterized in that: The segmentation module includes: An initial data volume determination submodule, used to determine the initial data volume of each region according to the performance of the target open source database; An optimization submodule, configured to perform an optimization operation on each of the regions according to the total amount of derivatives of the target data warehouse tool and the initial data amount, so as to obtain the number of optimal regions and the optimal data amount corresponding to each of the optimal regions; A primary key sorting submodule, used for extracting primary keys from the elastic distributed data set and sorting the primary keys to obtain KeyRdd; The submodule for obtaining the total number of partitions is used to obtain the total number of all partitions; A distribution quantity calculation submodule, used for calculating the uniform data distribution quantity according to the total quantity of all partitions and the optimal data quantity corresponding to each optimal area; The even-dividing submodule is used to evenly divide the elastic distributed data set according to the even data distribution quantity to obtain the even-cut boundary data.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the regional data allocation method applied to an open source database according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the regional data allocation method applied to an open source database according to any one of claims 1 to 6 are implemented.

Images