An optimization method for data lake long tail data

Abstract

The application discloses a kind of optimization methods of data lake long tail data, collect the data [1, T] in certain period;Fitting obtains the power law in this time;Obtain the split point: when the ratio of the data amount of [1, X] and total amount [1, T] is &,Then X is the split point in data;For the data before split point, new data is divided into update data and insert data, and is written into log file and base file respectively;For the data after split point, whether exist in base file or not, the data is forced to be written into the log file corresponding to the base file.The application accelerates the writing performance in the long tail task of MOR, can reduce more than 60% of index resources, for data lake query, small files are combined into large files, reduce data scanning amount, at the same time, data management is more reasonable, greatly optimize the speed of conditional query.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data lake, and particularly relates to a data lake long-tail data optimization method. BACKGROUND

[0002] In the field of data lake technology, one problem that cannot be avoided is the long-tail theorem. In the e-commerce field, the most common case is order status. If partitioning is established by day, the data status in the partition of the day's order data may be modified due to after-sales. The probability of after-sales modification decreases over time, but it is unknown when it will end. In data lake writing, modifying the status of this part still consumes high costs, and does not decrease because of the reduction of data volume.

[0003] The "long tail" is actually a colloquial expression of the power law and Pareto distribution characteristics in statistics, which is shaped like Figure 1 The protruding part in the normal curve is called "head", and the relatively flat part on the right is called "tail". People usually only pay attention to important people or important things, that is, they pay attention to the "head" of the curve, and ignore the "tail" of the curve. But in the network era, the overall benefit generated by paying attention to the "tail" may even exceed that of the "head". For data lake writing, in the long-tail case, more than 80% of resources will be consumed to process 20% of data. The present design mainly solves the problem of resource consumption caused by the long tail in the data lake.

[0004] In the data lake, taking Hudi as an example, two writing modes are provided, which are COW (Copy on Write) and MOR (Merge on Read). COW involves updating data each time it is written, copying historical data, and then merging it with new data to write a new data, Figure 2 is a schematic diagram of a single partition in a COW table. In the partition of the COW table, each time new data comes in, it will be merged with historical data, and read-write amplification will obviously exist, such as Figure 2 Data version 0 will be written to version 1, then to version 2, and finally to version N+1. After N times of read-write, read-write amplification obviously exists. This method is generally used in scenarios with less writing and more reading.

[0005] In order to solve the COW write amplification problem, the MOR format table appears, and the MOR compares the new data with the historical data every time, and splits into insert (when the same key data does not exist in the history) and upsert (when the same key data exists in the history) two kinds of data, and then writes into the base file and the log file. When the log file reaches a certain amount, the base and log files will also be merged to reduce the number of files, so as to speed up the query, and the specific process is as Figure 3 . Every time new data comes in, it will be compared with the base file in the partition, and according to the comparison result, it will be divided into update data and insert data, and the insert data will be appended to the base file, and the update will be written to the log file. This way reduces the write amplification of the base file, so it is more suitable for write-heavy scenarios than COW.

[0006] But it is worth noting that whether it is MOR or COW, there will be a Tag process, that is, every time you write, you need to compare with the historical data file, even if the new data is only one, it also needs to compare with the historical data in the file group in the partition to determine whether the data is insert or update. The Tag process mainly speeds up the comparison process through index, and the index is divided into complete index and incomplete index, such as the commonly used complete index based on hbase and incomplete index based on bloom. When the data writing tends to be stable, if this method is still used, a large amount of storage resources and computing resources will be consumed, especially for tasks that cannot be confirmed to end, the complete index must be held for a long time, consuming a large amount of storage resources, and for the incomplete index, a second check is required, consuming a large amount of computing resources. A large amount of resources exists in the necessary path, and the process is unreasonable. SUMMARY

[0007] Therefore, the present application optimizes and solves the resource consumption of the long-tail task in the MOR scenario, and provides a solution for the MOR scenario in the community, and improves the data lake optimization write scenario. In the patent application of a data lake optimization write method, a scheme for reducing resources and accelerating writing is introduced.

[0008] This scheme is not limited to the MOR scenario, and has the same effect on the TAG process of COW. The data flow is not limited to the power law distribution, and meets the scheme for any scenario that tends to be stable. Here, only the MOR in the long-tail scenario is used as a special case for implementation, and the design architecture is as Figure 7 . The present application provides a solution for this situation, which can release the index in advance in the long-tail scenario.

[0009] It should be noted that, regardless of MOR and COW are based on partition read and write data, so the application analyzes and improves the scene in a partition, and the whole table is composed of partitions.

[0010] The application discloses a long-tail data optimization method in a data lake, comprising the following steps:

[0011] Collect data [1, T] in a certain period;

[0012] Fit the power law in this period;

[0013] Find the split point: when the ratio of the data amount [1, X] to the total amount [1, T] is &, then X is the split point in the data;

[0014] For the data before the split point, the new data is divided into update data and insert data, and is written into the log file and the base file respectively;

[0015] For a certain data after the split point, if it does not exist in the existing base file, the data is forced to be written into the log file.

[0016] Further, the mathematical expression of the power law distribution is:

[0017] ;

[0018] Take the logarithm of both sides to get:

[0019] lgY=lga-blgX

[0020] Let y=lgY, x=lgX, and c=lga, then:

[0021] y=c-bx

[0022] Where c and b are the data increment of each day in a period of time, and then the least square method is used to obtain, and then Obtain a, and finally obtain the power law distribution equation.

[0023] Further, the split point is calculated as follows:

[0024] ;

[0025] Where & is the confidence space, T is the data period length, and b is the exponential coefficient.

[0026] Further, the total amount of data in the data period length T is as follows:

[0027] ;

[0028] ;

[0029] The inflection point is determined by the ratio of the data amount of [1, X] to the total amount [1, T], that is, after the data of & is written, it is considered that the inflection point has been reached.

[0030] Further, each piece of data is regrouped so that each piece of data determines a unique belonging base file.

[0031] Further, the grouping method comprises a hash method and a range-based grouping.

[0032] Further, after grouping, no new base file is generated in the partition, and all data is written into the corresponding log file, and the left join method is used when reading data; the system combines the base file and the log file at regular intervals.

[0033] Further, if the data amount in the partition tends to be stable and the data amount is significantly reduced, the data in the partition is regrouped again, and the full amount of data in the partition is required during partitioning.

[0034] The data is grouped according to rules, and each group has and only has one base file, and no new base file is added subsequently;

[0035] After grouping, newly written data is judged by rules to belong to which group, and then all the data is written into the log file of the group, and the base file and the log file are combined at regular intervals to speed up the query;

[0036] The new grouped base file will construct a bloom index for speeding up the query.

[0037] Further, the new writing process no longer needs to do the tag process, if the index writing is based on complete index, after the partition succeeds, the partition index is deleted, and resources can be released; if the index writing is in a non-complete index mode, the filtering based on bloom is reduced, and the second verification is also reduced, so that the data writing process is accelerated.

[0038] The beneficial effects of the present application are as follows:

[0039] The present application solves the long-tail task of MOR and accelerates the writing performance, and for complete index, more than 60% of index resources can be reduced, and for non-complete index, large-scale scanning calculation can also be reduced by 60%.

[0040] For data lake query, small files can be combined into large files to reduce data scanning amount, and data management is more reasonable, so that the speed of conditional query is greatly optimized. DETAILED DESCRIPTION

[0041] Figure 1 Long tail distribution class diagram

[0042] Figure 2 Hudi COW write flow diagram

[0043] Figure 3 Hudi MOR write flow diagram

[0044] Figure 4 Data distribution diagram

[0045] Figure 5 Data write scenario example

[0046] Figure 6 Data from the new allocation example diagram

[0047] Figure 7 MOR special processing diagram of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described below in conjunction with the accompanying drawings, but in no way limits the present application, any transformation or replacement based on the teaching of the present application, which belongs to the protection scope of the present application.

[0049] For the MOR scenario of community Hudi, long tail tasks will exist in the data writing process for a long time, which will cause serious read amplification and resource consumption. The present application can solve this problem, greatly accelerate the writing performance of tail data and reduce resource consumption.

[0050] After solving the long tail task, the present application also solves the problem that the index cannot be released, further reducing resource consumption.

[0051] The present application mainly provides two aspects of improvement.

[0052] 1) When does the long tail task perform long tail special processing.

[0053] 2) How to perform special processing.

[0054] For question 1, when does the long tail task start special processing, that is, find the time cutting point.

[0055] The long tail task is a kind of colloquial distribution with power law and Pareto distribution characteristics. Pareto has given the optimal solution in economics, and the 80-20 law of Pareto can give the inflection point. However, in engineering, the data is generally discretized by time, and the data is time effective. Assuming that the engineering data distribution is as follows Figure 4 .

[0056] Through the collection of a period of data (such as the previous 7 days, 14 days, etc.), then the power law is fitted to obtain the segmentation point, and finally the data after the segmentation point will be specially processed. The following is the simulation data, just to facilitate the derivation process. The data is limited to 1 month of validity, the actual application is subject to business, but it must be time-limited, this embodiment does not limit it. The specific solving process is as follows.

[0057] First step: obtain the data distribution law, and fit the power law derivation process

[0058] Mathematical expression of power law distribution:

[0059] ;

[0060] Two steps to obtain the logarithm of 10

[0061] lgY=lga-blgX

[0062] Let y=lgY, x=lgX, c=lga, then

[0063] y=c-bx

[0064] According to the day-by-day statistics of the data increment in a week, then the least square method is used to obtain c and b, and through Obtain a, and finally obtain the power law distribution equation.

[0065] Second step, obtain the segmentation point, and the derivation process is as follows:

[0066] 1. Obtain the data amount in the whole period, which is the total amount of data from 1 to T (data validity period)

[0067] ;

[0068] ;

[0069] 2. Obtain the segmentation point, so that the ratio of the data amount of [1, X] to the total amount of [1, T] is &. That is, after the data of & is written, it is considered that the inflection point has been reached. Special processing can be used;

[0070] ;

[0071] ;

[0072] ;

[0073] Among them, & is the confidence space, generally taking 90%, T is the data period length, generally taking 3 years 356*3 days, b is the exponential coefficient.

[0074] The split point X can be obtained through the above formula, and the data after the split point X can be processed specially. However, further constraints will also be made in engineering, and in some embodiments, when the write data in the partition is obviously reduced and tends to be stable, such as when the daily data amount is less than 1000, the special processing is directly entered.

[0075] Third step, data update: the purpose of hudi, whether COW or MOR, is ultimately to solve the update of data.

[0076] COW solves the update and delete process when writing, and the specific process is as follows: when writing, the historical data is copied each time, and then the new data is merged to cover the existing data, so as to achieve the purpose of updating. Because of the multiple copy base files, it causes serious read-write amplification. Therefore, it is not suitable for writing multiple scenarios.

[0077] The existing MOR solves the update and delete process when reading data, and the specific process is as follows: first, the new data is divided into upsert data and insert data according to the historical data, upsert is written to the new log file, and insert is appended to the base file. Each time is sequential writing, which can solve the write amplification problem. When reading, the log data is left joined with the base data to achieve the purpose of updating, and the log file and the base file are also merged at regular intervals to speed up the reading process.

[0078] The present application is improved based on the current reading method, such as Figure 5 The three scenarios are as follows:

[0079] Scenario 1: there is already a base file, new data k=2, v=5 (key is the primary key, value is other value), because k=2 exists in the base file, write to the log file;

[0080] Scenario 2: there is already a base file, new data k=2, v=5, but there is no k=2 in the base file, append the new data to the base file;

[0081] Scenario 3: there is already a base file, new data k=2, v=5, although k=2 does not exist in the base file, but unlike the prior art, the present application still forces the new data to be written to the log file.

[0082] The results of the query are the same in the above three scenarios. In comparison between scenario 1 and scenario 3, regardless of whether data exists in the base file, the data is written into the log file, and the data can still be normally queried. In this way, regardless of whether a record exists in the base file, the new record is written into the log file, and the insert and upsert of the data can still be implemented, the scanning of the base file is reduced, and the index can be released in advance.

[0083] Why does the application force the new data to be written into the log file without affecting the query speed? The reason is that the query is performed by left join, and if the log file exists, the log file is used as the main file, and if the log file does not exist, the base file is used as the main file.

[0084] Why does the application not remove the base file mode and use the log file mode? The reason is that the query is performed by left join, and the larger the log file, the slower the query efficiency, which is not desirable. Therefore, the application is only suitable for small data processing, such as long tail data, tail data that tends to be stable, and scenarios in which a few data are modified from time to time.

[0085] The community solution cannot directly use the application at present, because there is generally more than one base file in the same partition, and there is no rule between the bases. If the index is not used, it is impossible to determine in which base file the data is located, so the application further improves the community solution by performing global repartitioning in the partition. Preferably, the application uses two rules for hash-based or range-based grouping. Figure 6 Taking the hash-based rule as an example:

[0086] Figure 6 The hash rule is used for re-grouping, and after re-grouping, each data can determine a unique belonging file. For example, the hash (key) of the newly written data is 123, the number of files after re-partitioning is 10, and the data is written into the fourth (123%10+1=4) file. Therefore, through the repartitioning scheme, each data can find the corresponding base file, and the base file will construct a bloom index and a zorder index during repartitioning to speed up the query. After repartitioning, no new base file will be generated in the partition, and all data will be written into the corresponding log file, and the left join method is used during reading. The system will also periodically merge the base file and the log file, but this merging will not change the grouping rule. As the size of the single base file increases, repartitioning can be performed again.

[0087] In summary, in the MOR architecture of the community, the statistics of the daily increment can be increased, and the data can be fitted, and then the split point can be calculated by the formula, or the number of data can be increased, or the threshold value can be set to calculate the split point. The data before the split point is normally processed, and the new data is divided into update and insert and written into the log file and the base file respectively. The data after the split point is not scanned by the base, and is directly written into the log file. Specifically, as shown in Figure 7

[0088] For the complete index mode, the index can be removed after the split point, and there is no need to judge whether it is update or insert through the index. All data is written into the log file. For the non-complete index, the tag process is also reduced, and all data is logged into the log file.

[0089] For the COW scenario, after repartitioning, each batch of data can quickly find the corresponding base file, and then compare it through the bloom index. If all the data is insert data, it can be directly appended. If there is upsert data, it can be directly copied and merged.

[0090] When the data volume is less than a certain value, fitting is not needed, and the optimized method processing flow can be directly entered. If the business can predict the cutting point, fitting can also be omitted.

[0091] In one embodiment, the present application can be applied to the MOR storage of the community version of hudi. In the MOR scenario, the community also needs to go through the Tag service. The hudi community has several index modes for tag, which are generally divided into incomplete index and complete index, such as bloom filter-based incomplete index and hbase-based complete index. Incomplete index needs to be confirmed twice, and complete index needs to store full historical data. The present application can reduce the tag process for long-tail data, improve efficiency, and release index data in advance for the complete index mode, saving resources and improving efficiency.

[0092] ​For each partition, normally, new data is divided into update and insert and written into log files and base files. As the data in the partition decreases significantly over time, when the threshold is reached, the partition will first be repartitioned (partitioned). By default, the repartitioning is based on the range rule. After repartitioning, no new base file will be generated in the partition. Subsequent new data will be based on the partition rule to find the corresponding base file, and then all the data will be written into the corresponding log file. When the log file reaches a certain amount, the base file and the log file will be combined into a row file. The subsequent process will not perform a tag process, which speeds up the writing process. For complete index, the subsequent index can be released to achieve the purpose of saving resources.

[0093] The repartitioning process and the community clustering based on the primary key sorting process are similar, and can greatly reduce the query time. The main reason is that after the data is repartitioned, the positioning function is accelerated, and the file scanning is reduced.

[0094] In some embodiments, for the Chinese patent "A data lake optimization writing method" with application number 2022104974996, it also belongs to the complete index mode, so it can also apply the present application to quickly release index resources and achieve the purpose of saving costs.

[0095] In the MOR architecture of the open source hudi, each partition can partition the statistics of the daily increment of data, and fit the data according to the partition. Then, the partitioning point is calculated by formula. The data before the partitioning point is normally processed, and new data is divided into update and insert and written into log files and base files.

[0096] In some embodiments, if the amount of data after the partitioning point in the partition is small, a new way of processing is adopted. First, the data in the partition is repartitioned. This function is similar to the community's Clustering, but not exactly the same. The repartitioning must involve all the data in the partition, and the time point must be when the data in the warehouse tends to be stable. The data is grouped according to the rules. Each group has and only has one base, and no new base file will be added in the future.

[0097] After grouping, newly written data only needs to be judged by rules to determine which group the data belongs to, and then all the data is written into the log file of the group. When querying, because it is based on a left join-like method, it has no effect on the query result and does not conflict with the query of the existing community. The base file and the log file in each group will also be combined at regular intervals to speed up the query. The new grouped base file will still build a bloom filter to speed up the query.

[0098] Because the new write process no longer needs to do the tag process, if the write to the index is based on a complete index, the partition index can be deleted after the repartition is successful, releasing resources. For the non-complete index method, only the "non-existence" can be accurately judged, and the "existence" cannot be accurately judged. If the bloom judgment is "existence", a second check is still needed with the corresponding base file full file scan, i.e. secondary verification. The present application can reduce the secondary verification based on bloom for the non-complete index method, such as bloom, and speed up the data write process.

[0099] In the prior art, the community has a Tag stage when writing data, whether it is MOR or COW. The Tag stage is based on an index. The community has a complete index, such as an index based on Hbase, and also has an incomplete index, such as an index based on Bloom. The incomplete index can only accurately judge "no" and cannot accurately judge "yes". For "yes", a second judgment is needed. The base file needs to be scanned twice.

[0100] The ultimate goal of the present application is to release the index data of the complete index or the secondary scanning process of the non-complete index at the appropriate position. The community's solution cannot completely release the data of the index. Unless the business is 100% sure that there is no new data. After repartitioning, the data organization is more reasonable, and the performance of conditional queries is significantly improved.

[0101] In some embodiments, for general daily incremental data less than 10,000 and not excessively increased thereafter, it is considered to be at an appropriate position. Repartitioning at the appropriate position can reduce the left table of the left join, i.e. the amount of log data. Too much log data will affect the query, and less log data can speed up the query because of reasonable partitioning.

[0102] The present application solves the long-tail task of MOR and speeds up the write performance and reduces resource consumption. The following are the experimental results. For a long-tail task, under normal circumstances, it can be reduced to an appropriate position in 7 days, and the write performance is improved by 50%. Figure 5The data of simulation is taken as an example, it is known that T=30, &=90%, the simulation calculation of synchronization obtains b=0.9, then X=8.937 can be obtained by the following calculation of the following formula, and 9 is obtained by taking 1 method;

[0103] ;

[0104] Therefore, the optimization method of the application is used to solve the following 20 days from the 9th day, and 60% of the index resources can be reduced, and the community mode can also reduce 60% of the large-scale scanning calculation.

[0105] For data lake query, the function of Repartition based on the sort range rule is similar to the function of clustering based on the sort of the community, both of which can combine small files into large files, and can reduce data scanning amount, and data management is more reasonable, and for conditional query, great optimization is obtained.

[0106] The beneficial effects of the application are as follows:

[0107] The application solves the tail process of MOR and COW in the task, accelerates the writing performance, and can reduce more than 60% of index resources for complete index, and can also reduce 60% of large-scale scanning calculation for non-complete index;

[0108] For data lake query, small files can be combined into large files, data scanning amount is reduced, data management is more reasonable, and the speed of conditional query is greatly optimized;

[0109] The word "preferred" used in this text means used as an example, example or illustration. Any aspect or design described as "preferred" in this text does not have to be interpreted as more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" aims to propose the concept in a specific way. The term "or" as used in this application is intended to mean inclusive "or" rather than exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the permutations. That is, if X uses A; X uses B; or X uses both A and B, "X uses A or B" is satisfied in any of the foregoing examples.

[0110] Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and accompanying drawings. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), whether or not the component is structurally identical to the disclosed structure of the exemplary implementations of the present disclosure. In addition, while a particular feature of the present disclosure can have been disclosed with respect to only one of several implementations, such feature can be combined with one or other features of the other implementations as can be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "including", "includes", "having", "has", "contain", "contains", or variants thereof to be afforded like meanings in the context of this specification are used in the detailed description and / or in the claims, such terms are intended to be inclusive in a manner similar to the term "comprising" as an opening

[0111] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can exist physically, or a plurality of or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software functional module. If the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. The above-mentioned devices or systems can execute the storage method in the corresponding method embodiments.

[0112] In summary, the above-mentioned embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above-mentioned embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present application are equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. An optimization method for data lake long tail data, characterized in that, The method comprises the following steps: Collecting data in a certain period [1, T], T being the length of the data period; Fitting to obtain the power-law distribution in this period, the power-law distribution being: ； Taking the logarithm of both sides to obtain: lgY = lga - b lgX Let y = lgY, x = lgX, and c = lga to obtain: y = c - bx Where c and b are calculated by the least square method from the daily data increment in a period of time, and then through Where c and b are calculated by the least square method from the daily data increment in a period of time, and then through Where c and b are calculated by the least square method from the daily data increment in a period of time, and then through Obtaining the division point: when the ratio of the data amount in [1, X] to the total amount [1, T] is &, X is the division point in the data, and & is the confidence space; For the data before the division point, the new data is divided into update data and insert data, which are written into the log file and the base file, respectively; For the data after the division point, the data is forced to be written into the log file regardless of whether the data exists in the existing base file, and each piece of data is regrouped so as to determine the unique base file to which the data belongs; After grouping, no new base file is generated in the partition, and all data is written into the corresponding log file, and the left join method is used when reading data; the system combines the grouped base file and the log file at a fixed time, and the base file is partitioned again when the amount of the base file reaches a certain amount.

2. The method of claim 1, wherein, The division point is calculated as follows: 。 3. The method of claim 1, wherein, The total amount of data in the data period T is as follows: ； ； Obtaining the division point so that the ratio of the data amount in [1, X] to the total amount [1, T] is &, that is, after & data is written, it is considered that the inflection point has been reached.

4. The method of claim 3, wherein, The grouping method comprises a hash method and a range-based grouping.

5. The method of claim 1, wherein, If the amount of data in the partition tends to be stable and the amount of data is significantly reduced, the data in the partition is regrouped, and the total amount of data in the partition is required during partitioning; Grouping the data according to the rules, each group has and only has one base file, and no new base file is added subsequently; After grouping, the newly written data is written into the log file of the group to which the data belongs according to the rules, and the base file and the log file are combined at a fixed time to speed up the query; The new grouped base file constructs a bloom filter for speeding up the query.

6. The method of claim 5, wherein, The new writing process no longer needs to do the tag process, if the writing to the index is based on the complete index, the partition index is deleted after the partition is successful to release the resources; if the writing to the index is in a non-complete index manner, the bloom-based secondary check is reduced to speed up the data writing process.

Images