Data tilt correction method, device and computer readable storage medium

Abstract

The present invention relates to artificial intelligence technology, and discloses a data skew correction method, including: receiving a data skew correction instruction input by a user, extracting an original data set from multiple servers according to the data skew correction instruction, performing data amplification processing and adding random number processing on the original data set to obtain a data set to be partitioned, performing a partition operation on the data set to be partitioned to obtain a partitioned data set, performing statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set, performing data skew correction on the tuple partitioned data set using a partition correction algorithm and redistributing it to the multiple servers to complete the data skew correction. The present invention also proposes a data skew correction device and a computer-readable storage medium. The present invention can realize an accurate and efficient data skew correction function.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and computer-readable storage medium for data tilt correction. Background Art

[0002] For cluster systems, cache is generally distributed, that is, different server nodes are responsible for a certain range of cache data. We do not disperse cache data enough, resulting in a large amount of cache data being concentrated on one or several server nodes, which is called data skew. Current methods for correcting data skew include increasing jvm memory, increasing the number of data skew points, custom partitioning, redesigning keys, and using combiner merging. Although these methods have a certain effect on correcting data skew, these methods do not pre-process the data in advance, and because the data gap between tables is large, the processing process is more complicated, and there is no algorithm optimization. In the case of insufficient system memory, the efficiency of data skew correction is greatly affected. Summary of the invention

[0003] The present invention provides a data skew correction method, device and computer-readable storage medium, the main purpose of which is to perform data skew correction according to a partition correction algorithm, alleviate system computing pressure, and reduce system memory occupancy.

[0004] To achieve the above object, the present invention provides a data tilt correction method, comprising:

[0005] Receive a data skew correction instruction input by a user, extract original data sets from multiple servers according to the data skew correction instruction, perform data amplification processing and add random number processing on the original data sets, and obtain a data set to be partitioned;

[0006] Performing a partition operation on the data set to be partitioned to obtain a partitioned data set;

[0007] Performing statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set;

[0008] The partition correction algorithm is used to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

[0009] Optionally, performing a partition operation on the to-be-partitioned data set to obtain a partitioned data set includes:

[0010] Selecting a partitioning strategy for the data set to be partitioned;

[0011] Solving the constraint conditions of the partition operation according to the partition strategy;

[0012] According to the partitioning strategy and the constraint conditions, the data quantity of the partition is solved, and based on the data quantity, the data set to be partitioned is partitioned to obtain a partitioned data set.

[0013] Optionally, the performing statistics on the partitioned data set based on the pre-built tuple evaluation model to obtain the tuple partitioned data set includes:

[0014] Creating a tuple vector set corresponding to the partitioned data set;

[0015] Accumulating the tuple vector sets to obtain a global tuple vector set;

[0016] A tuple allocation vector is pre-constructed, and the global tuple vector set is statistically analyzed based on the tuple allocation vector to obtain the tuple partition data set.

[0017] Optionally, using a partition correction algorithm to correct data skew on the tuple partition data set and redistribute it to the multiple servers includes:

[0018] According to the storage space greedy strategy, the storage space of the existing data sets in the multiple servers is reallocated to obtain multiple servers with balanced storage space;

[0019] Storing the tuple partition data in the tuple partition data set in one of the servers with balanced storage space, and removing the server storing the tuple partition data from the multiple servers with balanced storage space, until the multiple servers with balanced storage space all store the tuple partition data to obtain multiple tuple partition data storage servers;

[0020] Determine whether there is any tuple partition data in the tuple partition data set; if there is any tuple partition data in the tuple partition data set, store the tuple partition data in one of the multiple tuple partition data storage servers; if there is no tuple partition data in the tuple partition data set, complete data tilting.

[0021] Optionally, the pre-built tuple allocation vector is:

[0022] AB=(R1,R2,R3,R b …, R r )

[0023] Wherein, AB is the pre-built tuple allocation vector, r is the number of scheduled tasks for data skew, and R is the number of tuples currently allocated;

[0024] R b =(f1, f2, f3, f i …, f n )(b∈[1,r])

[0025] R b The vector is the number of tuple allocation vectors allocated to the data skew, where f i Indicates the number of allocated tuple allocation vectors, the f i The calculation method is:

[0026]

[0027] Wherein, σ is the kernel number of the Gaussian kernel function, e is an infinite non-repeating decimal, α k is the tuple vector set, and k is the vector number of the tuple vector set.

[0028] In addition, to achieve the above-mentioned purpose, the present invention further provides a data skew correction device, which includes a memory and a processor, wherein the memory stores a data skew correction program that can be run on the processor, and when the data skew correction program is executed by the processor, the following steps are implemented:

[0029] Receive a data tilt correction instruction input by a user, extract original data sets from multiple servers, perform data amplification processing and add random number processing on the original data sets, and obtain a data set to be partitioned;

[0030] Performing a partition operation on the data set to be partitioned to obtain a partitioned data set;

[0031] Performing statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set;

[0032] The partition correction algorithm is used to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

[0033] Optionally, performing a partition operation on the to-be-partitioned data set to obtain a partitioned data set includes:

[0034] Selecting a partitioning strategy for the data set to be partitioned;

[0035] Solving the constraint conditions of the partition operation according to the partition strategy;

[0036] According to the partitioning strategy and the constraint conditions, the data quantity of the partition is solved, and based on the data quantity, the data set to be partitioned is partitioned to obtain a partitioned data set.

[0037] Optionally, the performing statistics on the partitioned data set based on the pre-built tuple evaluation model to obtain the tuple partitioned data set includes:

[0038] Creating a tuple vector set corresponding to the partitioned data set;

[0039] Accumulating the tuple vector sets to obtain a global tuple vector set;

[0040] A tuple allocation vector is pre-constructed, and the global tuple vector set is statistically analyzed based on the tuple allocation vector to obtain the tuple partition data set.

[0041] Optionally, using a partition correction algorithm to correct data skew on the tuple partition data set and redistribute it to the multiple servers includes:

[0042] According to the storage space greedy strategy, the storage space of the existing data sets in the multiple servers is reallocated to obtain multiple servers with balanced storage space;

[0043] Storing the tuple partition data in the tuple partition data set in one of the servers with balanced storage space, and removing the server storing the tuple partition data from the multiple servers with balanced storage space, until the multiple servers with balanced storage space all store the tuple partition data to obtain multiple tuple partition data storage servers;

[0044] Determine whether there is any tuple partition data in the tuple partition data set; if there is any tuple partition data in the tuple partition data set, store the tuple partition data in one of the multiple tuple partition data storage servers; if there is no tuple partition data in the tuple partition data set, complete data tilting.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data skew correction program is stored. The data skew correction program can be executed by one or more processors to implement the steps of the data skew correction method as described above.

[0046] The present invention amplifies the data, adds random number processing, and constructs a partition operation, so that the data gap between tables is small, the subsequent processing process is relatively simple, and the system calculation pressure is reduced. At the same time, the data set is statistically analyzed according to the pre-constructed tuple evaluation model, the system memory occupancy rate is reduced, and the impact on the efficiency of data skew correction under insufficient memory conditions is reduced. In addition, when the partition correction algorithm has a small amount of calculation, the partition correction efficiency is also improved. Therefore, the data skew correction method, device, and computer-readable storage medium proposed in the present invention can realize an efficient data skew correction function. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of a data tilt correction method provided by an embodiment of the present invention;

[0048] Figure 2A schematic diagram of the internal structure of a data tilt correction device provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a module of a data tilt correction program in a data tilt correction device provided in an embodiment of the present invention;

[0050] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] The present invention provides a method for correcting data tilt. Figure 1 FIG. 1 is a flow chart of a data tilt correction method provided by an embodiment of the present invention. The method can be executed by a device, and the device can be implemented by software and / or hardware.

[0053] In this embodiment, the data tilt correction method includes:

[0054] S1. Receive a data skew correction instruction input by a user, extract original data sets from multiple servers according to the data skew correction instruction, perform data amplification processing and add random number processing on the original data sets to obtain a data set to be partitioned.

[0055] Preferably, the original data set may include an ordinary small sample data set, such as a table set, and may also include a large sample data set based on a big data platform. For example, in the insurance industry, Model Point is an important data analysis module. Since the amount of data in the table set that the Model Point analysis module needs to process is very large, it can be placed on the big data platform to improve efficiency. However, it is found during operation that when a large table is associated with a small table, due to the uneven data between the large table and the small table, the data will be tilted to a certain server, which is the data tilt, thereby affecting the operating efficiency, and even the program will report an error and cannot run.

[0056] In one embodiment of the present invention, the original data set includes two tables, A and B, where table A is a large table and table B is a small table. The data amplification process refers to amplifying the data of table B (small table) by 100 times and increasing the multiple field (from 1 to 100); the random number addition process refers to adding a random number field to table A, and the value of the field is a random number from 1 to 100, so that the data of table A (large table) is not increased and will not affect the overall data volume.

[0057] After the data amplification process and random number addition process, a data set to be partitioned is obtained.

[0058] S2. Perform a partition operation on the data set to be partitioned to obtain a partitioned data set.

[0059] Preferably, the data set to be partitioned is partitioned, including: selecting a partitioning strategy for the data set to be partitioned, solving the constraints of the partitioning operation according to the partitioning strategy, solving the number of data in the partition according to the partitioning strategy and the constraints, and partitioning the data set to be partitioned based on the number of data to obtain a partitioned data set.

[0060] There are multiple partitioning strategies, which can be selected according to the different data types of the data set to be partitioned. For example, the data set to be partitioned of the table set type can adopt an iterative data partitioning strategy, and the iterative data partitioning strategy can be based on the Mapper table architecture. The partitioning strategy includes first performing table splitting on the data set to be partitioned to obtain multiple fine-grained data blocks (abbreviated as FGB), creating indexes based on the fine-grained data blocks, merging the multiple fine-grained data blocks with the same tuples in the table to obtain a fine-grained set to be partitioned, and allocating the fine-grained set to be partitioned to one or more nodes of the Mapper table architecture to obtain the partitioned data set. Among them, each group of partitioned data sets corresponds to its own index value. The tuple refers to each row in the table (that is, each record in the database), where each column of the table represents an attribute.

[0061] There are many types of constraints, such as the non-repetitive constraint principle. The non-repetitive constraint principle, that is, the constraint condition of the partition, is that if the fine-grained set to be partitioned already exists on the current Mapper table architecture node, no new fine-grained set to be partitioned will be allocated.

[0062] In the determining of the amount of data for each partition, the amount refers to the amount when the to-be-partitioned data set is split each time.

[0063] S3. Perform statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set.

[0064] Preferably, the tuple evaluation model includes a tuple vector generation module, a global tuple vector generation module and a statistical module.

[0065] In detail, S3 includes: creating a tuple vector set corresponding to the partitioned data set, accumulating the tuple vector set to obtain a global tuple vector set, pre-building a tuple allocation vector, and statistically analyzing the global tuple vector set based on the tuple allocation vector to obtain the tuple partitioned data set.

[0066] The tuple vector set is:

[0067] α=(t1,t2,t3,t i ,…,tn )

[0068] where t i =(index,factor,seg_no,slot_no)(i∈[1,n]), index is the index of the partitioned data set, factor is the index factor of the partitioned data set, seg_no is the data number of the partitioned data set, slot_no is the slot number of the partitioned data set. If slot_no is -1, it means that the tuple of the partitioned data set has not been counted yet.

[0069] When performing the accumulation of the tuple vector set, when there are m Mapper table architectures running simultaneously, the tuple vector set α on each Mapper table architecture is accumulated. k Accumulate and get the global tuple vector set AM:

[0070] AM=∑α k (k∈[1,m])

[0071] At the end of each partition of the Mapper table architecture, the tuple vector set α k Send to a pre-built vector storage unit in the partition. When the partition of the data set to be partitioned is completed, the partition vector storage unit stores all the tuple vector sets α k By accumulating them into the two-dimensional global tuple vector set AM, the global tuple vector set after the end of the current partition can be obtained.

[0072] Since the data that easily causes data skew has a one-to-one correspondence with the tuple, the present invention can calculate the data skew amount by counting the number of partitioned tuples, that is, counting the global tuple vector set can realize the correction of data skew. Preferably, the present invention first defines a tuple allocation vector AB:

[0073] AB=(R1,R2,R3,…,R r )

[0074] Where r is the number of tasks scheduled for the data skew, and R is the number of tuples currently assigned.

[0075] R b =(f1, f2, f3, ..., f n )(b∈[1,r])

[0076] R b The vector is the number of tuple allocation vectors used to record the data skew allocation. i Indicates the number of tuples allocated at a data tilt. The received partition data set can be counted by the above method to obtain the tuple partition data set.

[0077] Furthermore, the f n The calculation method can be based on the Gaussian kernel function:

[0078]

[0079] Wherein, σ is the kernel number of the Gaussian kernel function (also called the adjustment factor), e is an infinite non-repeating decimal, α k is the tuple vector set, and k is the vector number of the tuple vector set.

[0080] S4. Use a partition correction algorithm to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

[0081] In detail, using a partition correction algorithm to correct data skew on the tuple partition data set and redistribute it to the multiple servers includes:

[0082] According to the storage space greedy strategy, the storage space of the existing data sets in the multiple servers is reallocated to obtain multiple servers with balanced storage space;

[0083] Storing the tuple partition data in the tuple partition data set in one of the servers with balanced storage space, and removing the server storing the tuple partition data from the multiple servers with balanced storage space, until the multiple servers with balanced storage space all store the tuple partition data to obtain multiple tuple partition data storage servers;

[0084] Determine whether there is any tuple partition data in the tuple partition data set; if there is any tuple partition data in the tuple partition data set, store the tuple partition data in one of the multiple tuple partition data storage servers; if there is no tuple partition data in the tuple partition data set, complete data tilting.

[0085] Before the partition correction algorithm is used, there are two tables A and B in the insurance industry. A is a large table with billions of records, and B is a small table with thousands of records. An example of the original table structure is as follows:

[0086] Table A

[0087]

[0088]

[0089] Table B

[0090] Insurance Type Variable 1 Variable 2 123 1 1 125 1 2 …

[0091] After Table A and Table B are associated by insurance type, it is found that there are many insurance types in 123 (only 3 are listed in the example, but there may be more in actual situation). This will cause the table-based calculation of 123 insurance types to be tilted to a certain server, causing the data tilt problem.

[0092] A preferred embodiment of the present invention utilizes the partition correction algorithm to first eliminate servers that have been currently allocated tuple partition data sets through the X.add(i) method, and selects servers that need to be allocated tuple partition data sets in this round from the eliminated servers, calculates the storage space in the servers in this round and the data volume of the tuple partition data sets, and after intelligent allocation based on the storage space and the data volume, re-updates the storage volume of the server until the allocation of the tuple partition data sets is completed.

[0093] After the above data skew correction, when table A is associated with table B, the data is evenly distributed on 100 nodes, and the data used on each node is the same, as shown below:

[0094]

[0095] Table B

[0096]

[0097]

[0098] This can solve the problem of data skew and improve operating efficiency. Furthermore, the 100 nodes can be in different servers respectively, such as 100 nodes in 7 servers, so the data in the 7 servers completes data skew correction.

[0099] The present invention also provides a data tilt correction device. Figure 2 FIG. 1 is a schematic diagram of the internal structure of a data tilt correction device provided in one embodiment of the present invention.

[0100] In this embodiment, the data tilt correction device 1 can be a PC (Personal Computer), or a terminal device such as a smart phone, a tablet computer, a portable computer, or a server, etc. The data tilt correction device 1 at least includes a memory 11, a processor 12, a communication bus 13, and a network interface 14.

[0101] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the data tilt correction device 1, such as a hard disk of the data tilt correction device 1. In other embodiments, the memory 11 can also be an external storage device of the data tilt correction device 1, such as a plug-in hard disk equipped on the data tilt correction device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 11 can also include both an internal storage unit and an external storage device of the data tilt correction device 1. The memory 11 can not only be used to store application software and various types of data installed in the data tilt correction device 1, such as the code of the data tilt correction program 01, etc., but also can be used to temporarily store data that has been output or is to be output.

[0102] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, and is used to run program codes stored in the memory 11 or process data, such as executing the data tilt correction program 01.

[0103] The communication bus 13 is used to realize the connection and communication between these components.

[0104] The network interface 14 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the device 1 and other electronic devices.

[0105] Optionally, the device 1 may also include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the data tilt correction device 1 and to display a visual user interface.

[0106] Figure 2 Only the data tilt correction device 1 having components 11-14 and the data tilt correction program 01 is shown. It can be understood by those skilled in the art that Figure 1The structure shown does not constitute a limitation on the data tilt correction device 1, and may include fewer or more components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0107] exist Figure 2 In the embodiment of the device 1 shown, the memory 11 stores a data tilt correction program 01; when the processor 12 executes the data tilt correction program 01 stored in the memory 11, the following steps are implemented:

[0108] Step 1: receiving a data skew correction instruction input by a user, extracting original data sets from multiple servers according to the data skew correction instruction, performing data amplification processing and adding random number processing on the original data sets to obtain data sets to be partitioned.

[0109] Preferably, the original data set may include an ordinary small sample data set, such as a table set, and may also include a large sample data set based on a big data platform. For example, in the insurance industry, Model Point is an important data analysis module. Since the amount of data in the table set that the Model Point analysis module needs to process is very large, it can be placed on the big data platform to improve efficiency. However, it is found during operation that when a large table is associated with a small table, due to the uneven data between the large table and the small table, the data will be tilted to a certain server, which is the data tilt, thereby affecting the operating efficiency, and even the program will report an error and cannot run.

[0110] In one embodiment of the present invention, the original data set includes two tables, A and B, where table A is a large table and table B is a small table. The data amplification process refers to amplifying the data of table B (small table) by 100 times and increasing the multiple field (from 1 to 100); the random number addition process refers to adding a random number field to table A, and the value of the field is a random number from 1 to 100, so that the data of table A (large table) is not increased and will not affect the overall data volume.

[0111] After the data amplification process and random number addition process, a data set to be partitioned is obtained.

[0112] Step 2: performing a partition operation on the data set to be partitioned to obtain a partitioned data set.

[0113] Preferably, the data set to be partitioned is partitioned, including: selecting a partitioning strategy for the data set to be partitioned, solving the constraints of the partitioning operation according to the partitioning strategy, solving the number of data in the partition according to the partitioning strategy and the constraints, and partitioning the data set to be partitioned based on the number of data to obtain a partitioned data set.

[0114] There are multiple partitioning strategies, which can be selected according to the different data types of the data set to be partitioned. For example, the data set to be partitioned of the table set type can adopt an iterative data partitioning strategy, and the iterative data partitioning strategy can be based on the Mapper table architecture. The partitioning strategy includes first performing table splitting on the data set to be partitioned to obtain multiple fine-grained data blocks (abbreviated as FGB), creating indexes based on the fine-grained data blocks, merging the multiple fine-grained data blocks with the same tuples in the table to obtain a fine-grained set to be partitioned, and allocating the fine-grained set to be partitioned to one or more nodes of the Mapper table architecture to obtain the partitioned data set. Among them, each group of partitioned data sets corresponds to its own index value. The tuple refers to each row in the table (that is, each record in the database), where each column of the table represents an attribute.

[0115] There are many types of constraints, such as the non-repetitive constraint principle. The non-repetitive constraint principle, that is, the constraint condition of the partition, is that if the fine-grained set to be partitioned already exists on the current Mapper table architecture node, no new fine-grained set to be partitioned will be allocated.

[0116] In the determining of the amount of data for each partition, the amount refers to the amount when the to-be-partitioned data set is split each time.

[0117] Step three: Perform statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set.

[0118] Preferably, the tuple evaluation model includes a tuple vector generation module, a global tuple vector generation module and a statistical module.

[0119] In detail, S3 includes: creating a tuple vector set corresponding to the partitioned data set, accumulating the tuple vector set to obtain a global tuple vector set, pre-building a tuple allocation vector, and statistically analyzing the global tuple vector set based on the tuple allocation vector to obtain the tuple partitioned data set.

[0120] The tuple vector set is:

[0121] α=(t1,t2,t3,t i ,…,t n )

[0122] where t i =(index,factor,seg_no,slot_no)(i∈[1,n]), index is the index of the partitioned data set, factor is the index factor of the partitioned data set, seg_no is the data number of the partitioned data set, slot_no is the slot number of the partitioned data set. If slot_no is -1, it means that the tuple of the partitioned data set has not been counted yet.

[0123] When performing the accumulation of the tuple vector set, when there are m Mapper table architectures running simultaneously, the tuple vector set α on each Mapper table architecture is accumulated. k Accumulate and get the global tuple vector set AM:

[0124] AM=∑α k (k∈[1,m])

[0125] At the end of each partition of the Mapper table architecture, the tuple vector set α k Send to a pre-built vector storage unit in the partition. When the partition of the data set to be partitioned is completed, the partition vector storage unit stores all the tuple vector sets α k By accumulating them into the two-dimensional global tuple vector set AM, the global tuple vector set after the end of the current partition can be obtained.

[0126] Since the data that easily causes data skew has a one-to-one correspondence with the tuple, the present invention can calculate the data skew amount by counting the number of partitioned tuples, that is, counting the global tuple vector set can realize the correction of data skew. Preferably, the present invention first defines a tuple allocation vector AB:

[0127] AB=(R1,R2,R3,…,R r )

[0128] Where r is the number of tasks scheduled for the data skew, and R is the number of tuples currently assigned.

[0129] R b =(f1, f2, f3, ..., f n )(b∈[1,r])

[0130] R b The vector is the number of tuple allocation vectors used to record the data skew allocation. i Indicates the number of tuples allocated at a data tilt. The received partition data set can be counted by the above method to obtain the tuple partition data set.

[0131] Furthermore, the f n The calculation method can be based on the Gaussian kernel function:

[0132]

[0133] Wherein, σ is the kernel number of the Gaussian kernel function (also called the adjustment factor), e is an infinite non-repeating decimal, α k is the tuple vector set, and k is the vector number of the tuple vector set.

[0134] Step 4: Use a partition correction algorithm to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

[0135] In detail, using a partition correction algorithm to correct data skew on the tuple partition data set and redistribute it to the multiple servers includes:

[0136] According to the storage space greedy strategy, the storage space of the existing data sets in the multiple servers is reallocated to obtain multiple servers with balanced storage space;

[0137] Storing the tuple partition data in the tuple partition data set in one of the servers with balanced storage space, and removing the server storing the tuple partition data from the multiple servers with balanced storage space, until the multiple servers with balanced storage space all store the tuple partition data to obtain multiple tuple partition data storage servers;

[0138] Determine whether there is any tuple partition data in the tuple partition data set; if there is any tuple partition data in the tuple partition data set, store the tuple partition data in one of the multiple tuple partition data storage servers; if there is no tuple partition data in the tuple partition data set, complete data tilting.

[0139] Before the partition correction algorithm is used, there are two tables A and B in the insurance industry. A is a large table with billions of records, and B is a small table with thousands of records. An example of the original table structure is as follows:

[0140] Table A

[0141] Policy Main Rider No. Insurance Type variable xxx 0 123 1 xxx 1 124 5 yyy 0 123 1 yyy 1 125 2 ggg 2 123 4 …

[0142] Table B

[0143] Insurance Type Variable 1 Variable 2 123 1 1 125 1 2 …

[0144] After Table A and Table B are associated by insurance type, it is found that there are many insurance types in 123 (only 3 are listed in the example, but there may be more in actual situation). This will cause the table-based calculation of 123 insurance types to be tilted to a certain server, causing the data tilt problem.

[0145] A preferred embodiment of the present invention utilizes the partition correction algorithm to first eliminate servers that have been currently allocated tuple partition data sets through the X.add(i) method, and selects servers that need to be allocated tuple partition data sets in this round from the eliminated servers, calculates the storage space in the servers in this round and the data volume of the tuple partition data sets, and after intelligent allocation based on the storage space and the data volume, re-updates the storage volume of the server until the allocation of the tuple partition data sets is completed.

[0146] After the above data skew correction, when table A is associated with table B, the data is evenly distributed on 100 nodes, and the data used on each node is the same, as shown below:

[0147]

[0148] Table B

[0149] Insurance Type Variable 1 Variable 2 multiple 123 1 1 1 125 1 2 1 123 1 1 2 125 1 2 2 123 1 1 3 125 1 2 3 123 1 1 4 125 1 2 4 … …

[0150] This can solve the problem of data skew and improve operating efficiency. Furthermore, the 100 nodes can be in different servers respectively, such as 100 nodes in 7 servers, so the data in the 7 servers completes data skew correction.

[0151] Optionally, in other embodiments, the data skew correction program can also be divided into one or more modules, one or more modules are stored in the memory 11, and are executed by one or more processors (processor 12 in this embodiment) to complete the present invention. The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, and is used to describe the execution process of the data skew correction program in the data skew correction device.

[0152] For example, refer to Figure 3 FIG. 1 is a schematic diagram of a program module of a data tilt correction program in an embodiment of a data tilt correction device of the present invention. In this embodiment, the data tilt correction program can be divided into a data receiving and processing module 10, a partition operation module 20, a statistics module 30, and a partition correction and output module 40. For example:

[0153] The data receiving and processing module 10 is used to receive a data tilt correction instruction input by a user, extract original data sets from multiple servers, perform data amplification processing and add random number processing on the original data sets, and obtain a data set to be partitioned.

[0154] The partition operation module 20 is used to perform a partition operation on the to-be-partitioned data set to obtain a partitioned data set.

[0155] The statistical module 30 is used to perform statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set.

[0156] The partition correction and output module 40 is used to: use a partition correction algorithm to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

[0157] The functions or operation steps implemented when the program modules such as the data receiving and processing module 10, the partition operation module 20, the statistical module 30, the partition correction and output module 40 are executed are substantially the same as those in the above-mentioned embodiment and will not be repeated here.

[0158] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a data skew correction program is stored. The data skew correction program can be executed by one or more processors to implement the following operations:

[0159] A data tilt correction instruction input by a user is received, original data sets are extracted from multiple servers, and data amplification processing and random number adding processing are performed on the original data sets to obtain data sets to be partitioned.

[0160] The to-be-partitioned data set is partitioned to obtain a partitioned data set.

[0161] The partitioned data set is statistically analyzed based on a pre-built tuple evaluation model to obtain a tuple partitioned data set.

[0162] The partition correction algorithm is used to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

[0163] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0165] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data tilt correction method, characterized in that: The method comprises: Receive a data skew correction instruction input by a user, extract original data sets from multiple servers according to the data skew correction instruction, perform data amplification processing and add random number processing on the original data sets, and obtain a data set to be partitioned; Performing a partition operation on the data set to be partitioned to obtain a partitioned data set; Performing statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set; Using a partition correction algorithm to correct data skew on the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction; The method of performing statistics on the partitioned data set based on the pre-constructed tuple evaluation model to obtain the tuple partitioned data set includes: creating a tuple vector set corresponding to the partitioned data set; accumulating the tuple vector set to obtain a global tuple vector set; pre-constructing a tuple allocation vector, and performing statistics on the global tuple vector set based on the tuple allocation vector to obtain the tuple partitioned data set; The method uses a partition correction algorithm to correct data skew of the tuple partition data set and reallocate it to the multiple servers, including: reallocating storage space for existing data sets in the multiple servers according to a storage space greedy strategy to obtain multiple servers with balanced storage space; storing tuple partition data in the tuple partition data set to one of the servers with balanced storage space, and removing servers that have stored tuple partition data from the multiple servers with balanced storage space, until all the servers with balanced storage space store tuple partition data to obtain multiple tuple partition data storage servers; judging whether there is still tuple partition data in the tuple partition data set, if there is still tuple partition data in the tuple partition data set, storing the tuple partition data to one of the multiple tuple partition data storage servers, and completing data skew if there is no tuple partition data in the tuple partition data set; The pre-built tuple allocation vector is: in, Allocate a vector for the pre-built tuple, where r is the number of scheduled tasks for data skew, and R is the number of tuples currently allocated; The vector is the number of tuple allocation vectors allocated to the data skew, where, Indicates the number of allocated tuples in the i-th allocated vector. The calculation method is: in, is the number of kernels of the Gaussian kernel function, is an infinite non-repeating decimal, is the tuple vector set, The vector number of the tuple-vector set.

2. The data tilt correction method according to claim 1, characterized in that: The step of performing a partitioning operation on the data set to be partitioned to obtain a partitioned data set includes: Selecting a partitioning strategy for the data set to be partitioned; Solving the constraint conditions of the partition operation according to the partition strategy; According to the partitioning strategy and the constraint conditions, the data quantity of the partition is solved, and based on the data quantity, the data set to be partitioned is partitioned to obtain a partitioned data set.

3. A data tilt correction device, used to implement the data tilt correction method according to any one of claims 1 to 2, characterized in that: The device includes a memory and a processor, wherein the memory stores a data tilt correction program that can be run on the processor, and when the data tilt correction program is executed by the processor, the following steps are implemented: Receive a data skew correction instruction input by a user, extract original data sets from multiple servers according to the data skew correction instruction, perform data amplification processing and add random number processing on the original data sets, and obtain a data set to be partitioned; Performing a partition operation on the data set to be partitioned to obtain a partitioned data set; Performing statistics on the partitioned data set based on a pre-built tuple evaluation model to obtain a tuple partitioned data set; The partition correction algorithm is used to correct the data skew of the tuple partition data set and redistribute it to the multiple servers to complete the data skew correction.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data skew correction program, which can be executed by one or more processors to implement the data skew correction method as described in any one of claims 1 to 2.

Images