Real-time data warehouse increment synchronization method based on HUDI data lake

Abstract

The invention discloses a real-time data warehouse increment synchronization method based on an HUDI data lake, and relates to the technical field of data synchronization of the HUDI data lake, and the method comprises the following steps: obtaining the finest time partition granularity of the HUDI data lake, and collecting a data warehouse increment size based on the corresponding time partition granularity to obtain initial data warehouse increment data; preprocessing the initial data warehouse incremental data, and carrying out incremental size analysis to obtain an optimal partition size; the size of a time partition of the HUDI data lake is adjusted according to the optimal partition size, the optimal bucket size is obtained, and optimal bucket information is obtained; adjusting the bucket size of the HUDI data lake according to the optimal bucket information, and carrying out real-time synchronization on the increment of the data warehouse; the method is used for solving the problem that when an existing HUDI data lake data synchronization technology carries out real-time synchronization on data warehouse incremental data of e-commerce, the partition size and the bucket size of an HUDI data lake cannot be dynamically adjusted according to historical data warehouse incremental conditions, and then real-time data warehouse incremental synchronization cannot be carried out.

Description

Technical Field

[0001] This invention relates to the field of HUDI data lake data synchronization technology, specifically a real-time data warehouse incremental synchronization method based on HUDI data lake. Background Technology

[0002] HUDI data lake data synchronization technology is a technical system that relies on the core features of HUDI data lake, such as incremental storage, partitioning and bucketing, and metadata management, to achieve efficient and consistent data transmission and dynamic updates from the source to the HUDI lake warehouse. It covers the entire process of data capture, cleaning and transformation, partitioning and bucketing according to HUDI specifications, incremental synchronization, and storage optimization.

[0003] Existing HUDI data lake synchronization technology often manually sets fixed partition and bucket sizes when synchronizing incremental data in e-commerce data warehouses in real time. These manually fixed partition and bucket sizes not only fail to adapt to the dynamic fluctuations in data warehouse incremental data but also severely incompatible with the underlying HDFS storage design logic, easily leading to a double loss of core real-time synchronization pain points and underlying I / O performance. Data warehouse incremental data fluctuates with business peaks and troughs, cycles, and scenarios. Fixed values ​​during off-peak periods result in empty partitions and sparse data in buckets, wasting storage and metadata resources. During peak periods, they generate a large number of small files, triggering HUDI's compaction tasks and Flink synchronization to compete for cluster resources, significantly increasing data persistence latency. Sudden incremental data increases can cause an exponential surge in small files, leading to synchronization task backlog, a sharp drop in throughput, or even restarts, completely destroying synchronization stability. Furthermore, fixed values ​​are highly prone to deviating from HDFS values. The mainstream block standard of 128MB / 256MB results in cross-block storage, turning sequential I / O into random I / O and significantly reducing scanning efficiency. It also leads to unreasonable bucket numbers, either causing bloated metadata and increased query latency, or excessively large buckets and query lag. Furthermore, the hierarchical adaptation relationship between partitions and buckets is broken, resulting in half-full buckets, empty buckets, etc., exacerbating uneven data distribution and further causing I / O overhead. Therefore, the existing HUDI data lake data synchronization technology cannot dynamically adjust the partition size and bucket size of the HUDI data lake based on historical data warehouse incremental data when performing real-time data warehouse incremental synchronization. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the prior art. It obtains initial incremental data by acquiring the finest time partition granularity of the HUDI data lake and collecting data warehouse increments based on the corresponding time partition granularity; preprocesses the data and analyzes the increment size to obtain the optimal partition size; adjusts the time partition size of the HUDI data lake and obtains the optimal bucket size; adjusts the bucket size of the HUDI data lake and performs real-time synchronization of the data warehouse increments. This addresses the problem that existing HUDI data lake data synchronization technologies cannot dynamically adjust the partition and bucket sizes of the HUDI data lake based on historical data warehouse increment data during real-time synchronization of incremental data in e-commerce data warehouses, and then perform real-time data warehouse increment synchronization.

[0005] To achieve the above objectives, this application provides a real-time incremental data warehouse synchronization method based on HUDI data lake, including the following steps: Obtain the finest time partition granularity of the HUDI data lake, and collect the data warehouse increment size based on the corresponding time partition granularity to obtain the initial data warehouse increment data; Preprocess the initial incremental data in the data warehouse and perform incremental size analysis to obtain the optimal partition size; Adjust the size of the time partition of the HUDI data lake according to the optimal partition size, and obtain the optimal bucket size to get the optimal bucket information; Adjust the bucket size of the HUDI data lake based on the optimal bucket information and synchronize the data warehouse increments in real time.

[0006] Furthermore, obtaining the finest time partition granularity of the HUDI data lake and collecting the data warehouse increment size based on the corresponding time partition granularity to obtain the initial data warehouse increment data includes the following sub-steps: The HUDI data lake of e-commerce is denoted as the first data lake. The time partition granularity of the first data lake is obtained, and the finest time partition granularity is obtained and denoted as the finest partition granularity. The step size and unit corresponding to the finest partition granularity are denoted as k1. The actual physical storage data volume generated in the pre-data entry synchronization link of the first data lake real-time data warehouse incremental synchronization link, which only contains valid data added or changed by the business, is recorded as the original data increment.

[0007] Furthermore, obtaining the finest time partition granularity of the HUDI data lake and collecting the data warehouse increment size based on the corresponding time partition granularity to obtain the initial data warehouse increment data also includes the following sub-steps: Using k1 as the sampling period, the sum of the original data increments within the sampling period is recorded as the unit granularity data increment; the unit granularity data increments of the first data lake are continuously collected in each sampling period and arranged in chronological order, and recorded as the original data warehouse increment data; Set the first reference duration to t1 days, and extract the portion collected in the most recent t1 days from the original data warehouse incremental data, which is recorded as the initial data warehouse incremental data.

[0008] Furthermore, the initial incremental data in the data warehouse is preprocessed, and the incremental size is analyzed to obtain the optimal partition size, including the following sub-steps: Arrange the initial data warehouse incremental data in chronological order and denote it as the first incremental sequence. Denote any unit granularity data increment in the first incremental sequence as the first increment AD. For the initial data warehouse incremental data, obtain the first quartile AQ1, the third quartile AQ3, and the interquartile range IQ; and use the 3σ principle to remove outliers in the initial data warehouse incremental data, and calculate the mean and standard deviation of the remaining part, which are denoted as the corrected mean AP and the corrected standard deviation AB, respectively. Let TL = max[(AQ1 - 1.5 × IQ), (AP - 2 × AB)], TU = min[(AQ3 + 1.5 × IQ), (AP + 2 × AB)], and denote [TL, TU] as the global fluctuation range.

[0009] Furthermore, preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size includes the following sub-steps: In the first increment sequence, with the first increment as the center, obtain the k2 unit granularity data increments that are closest to the first increment, and denot them as the neighboring reference increments of the first increment; calculate the mean CA and standard deviation CB of the neighboring reference increments, and denote [CA-k3×CB, CA+k3×CB] as the local fluctuation range of the first increment, where k2 is the set number and k3 is the set proportional coefficient; If the first increment is within the local fluctuation range, it is marked as a normal value; if the first increment is not within the local fluctuation range and is not within the global fluctuation range, it is marked as an abnormal value; if the first increment is not within the local fluctuation range but is within the global fluctuation range, it is marked as a suspected abnormal value.

[0010] Furthermore, preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size includes the following sub-steps: All suspected abnormal values ​​are fully acquired to obtain a set of suspected values. In the first increment sequence, with the first increment as the center, the k4 unit-granularity data increments closest to the first increment are obtained and denoted as the first increment neighborhood reference increments, where k4 is the set number. The mean DP of the domain reference increment is calculated, and [|AD-DP| / DP]×100% is calculated as the relative deviation of the first increment. The relative deviation of all normal values ​​is calculated repeatedly to obtain the set of normal deviations, and the relative deviation of all suspected abnormal values ​​is calculated. Calculate the mean EP and standard deviation EB of the normal deviation set, and denote EP + k5 × EB as the normal deviation threshold; where k5 is the set scaling factor. For any suspected abnormal value, if the corresponding relative deviation is greater than the normal deviation threshold, it is marked as an abnormal value; otherwise, it is marked as a normal value. Repeatedly retrieve all outliers from the initial incremental data in the data warehouse and remove them to obtain valid incremental data.

[0011] Furthermore, preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size includes the following sub-steps: Calculate the standard deviation YB of the effective incremental data, and set the partition size range to [DS, DL]; Set the neighbor radius to 0.8×YB and the minimum number of points to n. Perform density clustering on the effective incremental data to obtain multiple clusters. Calculate the proportion of data in each cluster to the total number of data in the effective incremental data, and record it as the cluster proportion. Clusters with a proportion greater than k6 within a cluster are designated as core clusters. If the proportion of any cluster is less than or equal to k6, then the k7 clusters with the largest proportion within a cluster are designated as core clusters. Here, k6 is the set proportion and k7 is the set number. Calculate the median of each core cluster, and then calculate the weighted average of the medians of all core clusters based on their proportion within the cluster. This average is denoted as the normal core increment VH.

[0012] Furthermore, preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size includes the following sub-steps: Arrange the effective incremental data in chronological order and denote it as the second incremental sequence; let ES = "DS / VH", EL = ("DL / VH", 30), and use [ES, EL] as the range of values ​​for the merging multiple, where the merging multiple is an integer; Let any merging multiple be E0. Starting from the beginning of the second increment sequence, calculate the sum of E0 consecutive unit-granularity data increments. If there are no E0 consecutive unit-granularity data increments, discard the result. This will give you the merging increment sequence corresponding to E0. VH×E0 is denoted as the corresponding candidate partition size SV. All corresponding merge increment sequences and candidate partition sizes are repeatedly obtained. For the candidate partition size SV corresponding to E0, calculate the standard deviation EB of the merged incremental sequence corresponding to E0; calculate (SV / DS)×(1-EB / YB), denoted as the merged return index of SV; Repeatedly obtain the merge yield index of all candidate partition sizes, and denote the candidate partition size with the largest merge yield index as the optimal partition size WS.

[0013] Furthermore, the size of the time partition of the HUDI data lake is adjusted according to the optimal partition size, and the optimal bucket size is obtained. The optimal bucket information includes the following sub-steps: Adjust the size of the finest time partition in the HUDI data lake to the optimal partition size; Set a candidate set of bucket sizes {128MB, 256MB}, calculate "WS / 128" and "WS / 256" respectively, and record them as the reasonable number of buckets F1 corresponding to 128MB and the reasonable number of buckets F2 corresponding to 256MB respectively; If both reasonable bucket numbers F1 and F2 are located in [1, DL / 256], then the bucket size corresponding to the smaller reasonable bucket number is taken as the optimal bucket size; if only one of F1 and F2 is located in [1, DL / 256], then the bucket size corresponding to the reasonable bucket number located in [1, DL / 256] is taken as the optimal bucket size.

[0014] Furthermore, adjusting the bucket size of the HUDI data lake based on the optimal bucketing information and performing real-time synchronization of data warehouse increments includes the following sub-steps: Adjust the bucket size of the time partition of the HUDI data lake to the optimal bucket size, set the adjustment period to T0, and periodically adjust the size of the time partition and bucket size of the HUDI data lake according to T0; Real-time data warehouse increments are captured, and corresponding data tables are created in HUDI according to the adjusted partition and bucket sizes. The newly added data is then placed in the corresponding positions in the data tables according to the rules, thus completing real-time data warehouse increment synchronization.

[0015] The beneficial effects of this invention are as follows: This invention obtains the finest time partition granularity of the HUDI data lake and collects the incremental data size of the data warehouse based on the corresponding time partition granularity to obtain initial incremental data of the data warehouse; it preprocesses the initial incremental data of the data warehouse and analyzes the incremental size to obtain the optimal partition size; it adjusts the size of the time partition of the HUDI data lake according to the optimal partition size and obtains the optimal bucket size to obtain the optimal bucket information; it adjusts the bucket size of the HUDI data lake according to the optimal bucket information and performs real-time synchronization of the incremental data of the data warehouse; when performing real-time synchronization of incremental data of the e-commerce data warehouse, the partition size and bucket size of the HUDI data lake can be dynamically adjusted according to the historical data warehouse incremental situation, and then the incremental data of the data warehouse is synchronized in real time, which improves the stability and rationality of the incremental data synchronization of the data warehouse. This invention utilizes both global and corresponding local fluctuation ranges to simultaneously filter data based on global and local distribution characteristics. This allows for more reasonable judgments on data with periodic or short-term trends, reducing the probability of misjudgment. It performs density clustering on effective incremental data, filters out core clusters, and uses a weighted median to obtain the normal core increment. Compared to traditional mean or median, density clustering better identifies multimodal distributions, resulting in more reliable representative increment values ​​and improving the reliability and accuracy of subsequent calculations. By calculating the merging benefit index, it directly quantifies the increment size, standard deviation, and benefit, selecting the candidate partition size with the highest merging benefit as the optimal partition size. This allows multiple small increments to be merged into more suitable physical units, reducing the number of small files, decreasing query latency during synchronization, reducing file system metadata pressure, and lowering compaction and small file merging costs. Attached Figure Description

[0016] Figure 1 This is a flowchart of the steps of the method of the present invention; Figure 2 This is a flowchart of the abnormal suspected value screening process of the present invention; Figure 3 This is a flowchart illustrating the process of obtaining the optimal partition size according to the present invention. Figure 4 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 As shown, this application provides a real-time incremental data warehouse synchronization method based on HUDI data lake, including the following steps: Step S1 involves obtaining the finest time partition granularity of the HUDI data lake and collecting the data warehouse increment size based on the corresponding time partition granularity to obtain the initial data warehouse increment data. Step S1 includes the following sub-steps: Step S101: Denote the HUDI data lake of the e-commerce as the first data lake, obtain the time partition granularity of the first data lake, and obtain the finest time partition granularity, which is denoted as the finest partition granularity. Also obtain the step size and unit corresponding to the finest partition granularity, which is denoted as k1. For example, if the time partition of the HUDI data lake is 1 day, 1 hour, and 3 minutes, then the finest time partition granularity is minutes, the step size is 3, and k1 is 3 minutes. In step S102, the actual physical storage data volume generated in the synchronization link before entering the lake in the real-time data warehouse of the first data lake, which only contains valid data of business addition or change, is recorded as the original data increment. After HUDI and other writes are written, compression, merging, indexing metadata, etc. will occur, resulting in the file size after entering the lake not completely corresponding one-to-one with the original business increment. Using the original data increment before entering the lake can more accurately reflect the generated data volume.

[0019] Step S103: Using k1 as the sampling period, the sum of the original data increments within the sampling period is recorded as the unit granularity data increment; the unit granularity data increments of the first data lake in each sampling period are continuously collected and arranged in chronological order, and recorded as the original data warehouse increment data; if the sampling period is inconsistent with the actual partition granularity, it will cause the unit granularity data increment to be inconsistent with the partition, affecting subsequent analysis and processing; Step S104: Set the first reference duration to t1 days, extract the portion collected in the most recent t1 days from the original data warehouse incremental data, and record it as the initial data warehouse incremental data; in this embodiment, t1=30 days, which can be flexibly set according to the actual application scenario, generally [20 days, 60 days], and selecting data from the most recent several days can reflect the data incremental characteristics of the current business; In practice, HUDI Data Lake is a transactional data lake warehouse platform built around a database kernel. It brings core database functions to traditional data lakes and aims to solve pain points such as the inability of traditional data lakes to efficiently support data updates, deletions and incremental processing, lack of transaction guarantees leading to data consistency problems, proliferation of small files affecting query performance, and difficulty in supporting batch processing and stream processing scenarios at the same time.

[0020] Step S2 involves preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size. Step S2 includes the following sub-steps: For step S201, please refer to... Figure 2 As shown, the initial data warehouse incremental data is arranged in chronological order and denoted as the first incremental sequence. The increment of any unit granularity data in the first incremental sequence is denoted as the first increment AD. Step S202: For the initial data warehouse incremental data, obtain the first quartile AQ1, the third quartile AQ3, and the interquartile range IQ; and use the 3σ principle to remove outliers from the initial data warehouse incremental data, and calculate the mean and standard deviation of the remaining part, which are denoted as the corrected mean AP and the corrected standard deviation AB, respectively; after removing temporary extreme values, recalculate the mean and standard deviation to avoid the original outliers raising the standard deviation, which would lead to the subsequent global normal range being too wide and causing anomalies to be missed. Step S203: Let TL = max[(AQ1 - 1.5 × IQ), (AP - 2 × AB)], take the maximum value of the two to avoid the threshold being too low, and TU = min[(AQ3 + 1.5 × IQ), (AP + 2 × AB)], take the minimum value of the two to avoid the threshold being too high. Let [TL, TU] be recorded as the global fluctuation range; the range of incremental data that conforms to the global business rules. If it exceeds the range, it is considered a suspected anomaly.

[0021] Step S204: In the first increment sequence, with the first increment as the center, obtain the k2 unit granularity data increments closest to the first increment, and record them as the reference increments near the first increment; calculate the mean CA and standard deviation CB of the neighboring reference increments, and record [CA-k3×CB, CA+k3×CB] as the local fluctuation range of the first increment, where k2 is the set number and k3 is the set proportional coefficient; in this embodiment, k2 is 20, generally [10, 30], k3=3, generally [2, 3.5], and can be flexibly set; Step S205: If the first increment is within the local fluctuation range, it is marked as a normal value; if the first increment is not within the local fluctuation range and not within the global fluctuation range, it is marked as an abnormal value; if the first increment is not within the local fluctuation range but is within the global fluctuation range, it is marked as a suspected abnormal value. The first incremental sequence often has a short-term trend or cyclical trend. The global scope may judge short-term normal fluctuations as abnormal, while the local fluctuation range can correct this situation through neighboring points. Combining the local and global perspectives can reduce the probability of misjudging short-term normal fluctuations as abnormal and missing true abnormalities.

[0022] Step S206: Fully acquire all suspected abnormal values ​​to obtain a set of suspected values; in the first increment sequence, take the first increment as the center and acquire the k4 unit granularity data increments closest to the first increment, which are denoted as the first increment neighborhood reference increments, where k4 is the set number; in this embodiment, k4=6, which can be flexibly set, and is generally [4, 10]; Step S207: Calculate the mean DP of the domain reference increment, calculate [|AD-DP| / DP]×100%, and record it as the relative deviation of the first increment. Repeat the calculation of the relative deviation of all normal values ​​to obtain the set of normal deviations, and calculate the relative deviation of all suspected abnormal values. Use relative values ​​instead of absolute values ​​to avoid interference from different business data volumes. The larger the relative deviation, the higher the degree of abruptness of the point compared with the surrounding data, and the more likely it is to be a real anomaly. Step S208: Calculate the mean EP and standard deviation EB of the normal deviation set, and record EP + k5 × EB as the normal deviation threshold; where k5 is the set proportional coefficient; in this embodiment, k5 = 3, which can be flexibly set, generally [2, 3.5]; the normal deviation set reflects the natural and unbiased normal fluctuation characteristics of the incremental data in the data warehouse. Step S209: For any suspected abnormal value, if the corresponding relative deviation is greater than the normal deviation threshold, it is marked as an abnormal value; otherwise, it is marked as a normal value. Step S210: Repeatedly obtain all outliers in the initial data warehouse incremental data and remove them. After completion, valid incremental data is obtained.

[0023] For example, if the first incremental sequence is {48, 52, 45, 55, 49, 205, 53, 47, 51, 46, 25, 54, 50, 49, 56, 58, 44, 57, 52, 50, 53, 48, 55, 47, 51, 210, 49, 54, 46, 52, 30, 57, 48, 50, 200, 55, 49, 53, 47, 56} in MB, then AQ1=48, AQ3=54, IQ=6. After removing outliers using the 3σ principle, AP=61, AB=45, TL=42, TU=60, and the global fluctuation range is [42, 60]. Then, calculate their respective local fluctuation ranges and filter out the suspected outliers as {25, 30} and the outliers as {200, 205, 210}. The relative deviation of 25 is calculated to be 50.0%, and the relative deviation of 30 is 55.9%. The mean of the normal deviation set is EP=3.3%, and the standard deviation is EB=1.1%. Therefore, the normal deviation threshold is 6.6%, so 25 and 30 are outliers.

[0024] Step S211: Calculate the standard deviation YB of the effective incremental data and set the partition size range to [DS, DL]. In this embodiment, DS=64MB. If the partition is smaller than this value, it is easy to form HUDI small files, which increases the merging and query overhead. DL=2048MB. If the partition is larger than this value, it is easy to cause HUDI query scan to lag and reduce read performance. It can also be adjusted appropriately. For step S212, please refer to... Figure 3 As shown, with the neighbor radius set to 0.8×YB and the minimum number of points to n, density clustering is performed on the effective incremental data to obtain multiple clusters. The proportion of data in each cluster to the total number of data in the effective incremental data is calculated and denoted as the cluster proportion. The effective incremental data, after removing outliers, directly reflects the true business patterns, and its data distribution will naturally aggregate into multiple clusters, with each cluster representing a normal incremental scenario. In this embodiment, n is 6, but it can be set flexibly. Step S213: Clusters with a proportion greater than k6 within a cluster are designated as core clusters. If the proportion of all clusters is not greater than k6, the clusters with the largest proportion of k7 within a cluster are designated as core clusters. Here, k6 is the set proportion and k7 is the set number. In this embodiment, k6 = 10% and k7 = 5. Core clusters represent common situations, and the data they contain represent the level of the most common unit granularity data increment. Step S214: Calculate the median of each core cluster, and obtain the weighted average of the medians of all core clusters based on the proportion within each cluster. This average is denoted as the normal core increment VH. In other words, calculate the product of each median and the corresponding proportion within each cluster, sum them up, and then divide by the sum of the proportions within all core clusters.

[0025] Step S215: Arrange the effective incremental data in chronological order and denote it as the second incremental sequence; let ES = [DS / VH], EL = ("DL / VH", 30), and use [ES, EL] as the range of values ​​for the merge multiple, where the merge multiple is an integer; ES is the minimum merge multiple, which is a positive integer. If the merge multiple is less than ES, the corresponding partition size must be less than 64MB, which is an invalid value; EL is the maximum merge multiple, which is the upper limit of the merge multiple. If the merge multiple is greater than EL, either the partition is at the critical value of excessively large files, or excessive merging occurs; Step S216: Denote any merging factor as E0. Starting from the beginning of the second increment sequence, calculate the sum of E0 consecutive unit-granularity data increments. If there are no E0 consecutive unit-granularity data increments, discard the result. Obtain the merged increment sequence corresponding to E0. Because the second increment sequence removes outliers, the data at some positions is not continuous. For example, the second increment sequence is part of {50, 50, 50, 61, 60, 70}, where outliers between 60 and 70 are removed. If the merging factor is 3, calculate up to 61 and 60. If there are no 3 consecutive unit-granularity data increments, discard the result of 61+60.

[0026] Step S217: Record VH×E0 as the corresponding candidate partition size SV, and repeatedly obtain all corresponding merge increment sequences and candidate partition sizes; Step S218: For the candidate partition size SV corresponding to E0, calculate the standard deviation EB of the merged incremental sequence corresponding to E0; calculate (SV / DS)×(1-EB / YB), denoted as the merged return index of SV; SV / DS reflects the degree to which the candidate partition size is far from the small file threshold. The larger the ratio, the larger the partition size is than 64MB. The lower the disk I / O and computational overhead of HUDI small file merging, the higher the storage regularity and efficiency. 1-EB / YB reflects the effect of controlling the fluctuation of the partition increment after merging. The closer the value is to 1, the smaller the fluctuation of the increment after merging. The lower the probability of small files and super-large files generated due to business fluctuations in subsequent production, thus ensuring the stability of partition size from the root. Step S219: Repeatedly obtain the merge benefit index of all candidate partition sizes, and record the candidate partition size with the largest merge benefit index as the optimal partition size WS. For example, if the standard deviation YB = 15, [DS, DL] is [64, 2048], and the normal core increment VH = 50 (in MB), then ES = 2, EL = 30, and the merging multiple is {2, 3, 4, ..., 29, 30}. Then calculate the corresponding merging return index. For example, the standard deviations EB corresponding to {2, 10, 20, 30} are {10.61, 4.74, 3.35, 2.74}, and the corresponding merging return index is {0.458, 5.344, 12.141, 19.148}. Therefore, the candidate partition size of 1500MB corresponding to 30 is the optimal partition size. In practice, a larger merge yield index indicates that the partition size performs better in terms of storage optimization and volatility control.

[0027] Step S3: Adjust the size of the time partition of the HUDI data lake according to the optimal partition size, and obtain the optimal bucket size to get the optimal bucket information; Step S3 includes the following sub-steps: Step S301: Adjust the size of the finest time partition of the HUDI data lake to the optimal partition size; Step S302: Set the candidate set of bucket size {128MB, 256MB}, and calculate "WS / 128" and "WS / 256" respectively, which are recorded as the reasonable number of buckets F1 for 128MB and the reasonable number of buckets F2 for 256MB. 128MB and 256MB are the general optimal values ​​for distributed storage, which are suitable for most Hudi scenarios and can greatly simplify the selection. Step S303: If both reasonable bucket numbers F1 and F2 are located in [1, DL / 256], then the bucket size corresponding to the smaller reasonable bucket number is taken as the optimal bucket size; if only one of F1 and F2 is located in [1, DL / 256], then the bucket size corresponding to the reasonable bucket number located in [1, DL / 256] is taken as the optimal bucket size; the fewer the number of buckets, the lower the metadata overhead, and the actual data volume of a single bucket is closer to the candidate value, resulting in better uniformity; In the specific implementation process, if neither F1 nor F2 is located in [1, DL / 256], it means that the double constraint of EL=("DL / VH", 30) did not take effect when calculating the merging multiple. If it is caused by a calculation error, it needs to be recalculated.

[0028] Step S4 involves adjusting the bucket size of the HUDI data lake based on the optimal bucketing information and synchronizing the data warehouse increments in real time. Step S4 includes the following sub-steps: Step S401: Adjust the bucket size of the time partition of the HUDI data lake to the optimal bucket size, set the adjustment period to T0, and periodically adjust the size of the time partition and the bucket size of the HUDI data lake according to T0; in this embodiment, T0=30 days, that is, the partition size and the bucket size are adjusted once every 30 days. Step S402: Capture data warehouse increments in real time, and create corresponding data tables in HUDI according to the adjusted partition size and bucket size. Then, place the newly added data in the corresponding positions in the data tables according to the rules to complete the real-time data warehouse increment synchronization. In practice, the adjustment of the time partition size and bucket size only applies to newly synchronized data, and does not need to be adjusted for historical data in the data lake.

[0029] Example 2, please refer to Figure 4 As shown, Figure 4 A schematic diagram of an electronic device is provided, which may include a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The memory stores computer-readable instructions, and the processor can call these instructions. When the processor executes a computer-readable instruction, it performs steps similar to those in the HUDI data lake-based real-time data warehouse incremental synchronization method to achieve the following functions: obtaining the finest time partition granularity of the HUDI data lake and collecting the data warehouse incremental size based on the corresponding time partition granularity to obtain initial data warehouse incremental data; preprocessing the initial data warehouse incremental data and performing incremental size analysis to obtain the optimal partition size; adjusting the time partition size of the HUDI data lake according to the optimal partition size and obtaining the optimal bucket size to obtain optimal bucket information; adjusting the bucket size of the HUDI data lake according to the optimal bucket information and performing real-time synchronization of the data warehouse incremental data.

[0030] Furthermore, when the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0031] Example 3: This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps of the real-time data warehouse incremental synchronization method based on the HUDI data lake to achieve the following functions: obtaining the finest time partition granularity of the HUDI data lake, and collecting the data warehouse incremental size based on the corresponding time partition granularity to obtain initial data warehouse incremental data; preprocessing the initial data warehouse incremental data and performing incremental size analysis to obtain the optimal partition size; adjusting the time partition size of the HUDI data lake according to the optimal partition size, and obtaining the optimal bucket size to obtain optimal bucket information; adjusting the bucket size of the HUDI data lake according to the optimal bucket information, and performing real-time synchronization of the data warehouse incremental data.

[0032] Based on the above description of the embodiments, the embodiments of the present invention can be provided as methods, systems, or computer program products. Based on this understanding, the above technical solutions, in essence or in terms of their contribution to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or certain parts of the embodiments.

[0033] In the embodiments provided in this application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces. The indirect coupling or communication connection between systems, modules, and units may be electrical, mechanical, or other forms.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A real-time incremental data warehouse synchronization method based on HUDI data lake, characterized in that, Includes the following steps: Obtain the finest time partition granularity of the HUDI data lake, and collect the data warehouse increment size based on the corresponding time partition granularity to obtain the initial data warehouse increment data; Preprocess the initial incremental data in the data warehouse and perform incremental size analysis to obtain the optimal partition size; Adjust the size of the time partition of the HUDI data lake according to the optimal partition size, and obtain the optimal bucket size to get the optimal bucket information; Adjust the bucket size of the HUDI data lake based on the optimal bucket information and synchronize the data warehouse increments in real time.

2. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 1, characterized in that, Obtaining the finest time partition granularity of the HUDI data lake and collecting the data warehouse incremental size based on the corresponding time partition granularity to obtain the initial data warehouse incremental data includes the following sub-steps: The HUDI data lake of e-commerce is denoted as the first data lake. The time partition granularity of the first data lake is obtained, and the finest time partition granularity is obtained and denoted as the finest partition granularity. The step size and unit corresponding to the finest partition granularity are denoted as k1. The actual physical storage data volume generated in the pre-data entry synchronization link of the first data lake real-time data warehouse incremental synchronization link, which only contains valid data added or changed by the business, is recorded as the original data increment.

3. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 2, characterized in that, Obtaining the finest time partition granularity of the HUDI data lake and collecting the data warehouse increment size based on the corresponding time partition granularity to obtain the initial data warehouse increment data also includes the following sub-steps: Using k1 as the sampling period, the sum of the original data increments within the sampling period is recorded as the unit granularity data increment; the unit granularity data increments of the first data lake are continuously collected in each sampling period and arranged in chronological order, and recorded as the original data warehouse increment data; Set the first reference duration to t1 days, and extract the portion collected in the most recent t1 days from the original data warehouse incremental data, which is recorded as the initial data warehouse incremental data.

4. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 3, characterized in that, Preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size includes the following sub-steps: Arrange the initial data warehouse incremental data in chronological order and denote it as the first incremental sequence. Denote any unit granularity data increment in the first incremental sequence as the first increment AD. For the initial data warehouse incremental data, obtain the first quartile AQ1, the third quartile AQ3, and the interquartile range IQ; and use the 3σ principle to remove outliers in the initial data warehouse incremental data, and calculate the mean and standard deviation of the remaining part, which are denoted as the corrected mean AP and the corrected standard deviation AB, respectively. Let TL = max[(AQ1 - 1.5 × IQ), (AP - 2 × AB)], TU = min[(AQ3 + 1.5 × IQ), (AP + 2 × AB)], and denote [TL, TU] as the global fluctuation range.

5. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 4, characterized in that, Preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size also includes the following sub-steps: In the first increment sequence, taking the first increment as the center, obtain the k2 unit granularity data increments that are closest to the first increment, and denot them as the neighboring reference increments of the first increment; Calculate the mean CA and standard deviation CB of the neighboring reference increments, and denote [CA-k3×CB, CA+k3×CB] as the local fluctuation range of the first increment, where k2 is the number set and k3 is the set proportional coefficient; If the first increment is within the local fluctuation range, it is marked as a normal value; if the first increment is not within the local fluctuation range and is not within the global fluctuation range, it is marked as an abnormal value; if the first increment is not within the local fluctuation range but is within the global fluctuation range, it is marked as a suspected abnormal value.

6. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 5, characterized in that, Preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size also includes the following sub-steps: All suspected abnormal values ​​are fully acquired to obtain a set of suspected values. In the first increment sequence, with the first increment as the center, the k4 unit-granularity data increments closest to the first increment are obtained and denoted as the first increment neighborhood reference increments, where k4 is the set number. The mean DP of the domain reference increment is calculated, and [|AD-DP| / DP]×100% is calculated as the relative deviation of the first increment. The relative deviation of all normal values ​​is calculated repeatedly to obtain the set of normal deviations, and the relative deviation of all suspected abnormal values ​​is calculated. Calculate the mean EP and standard deviation EB of the normal deviation set, and denote EP + k5 × EB as the normal deviation threshold; where k5 is the set scaling factor. For any suspected abnormal value, if the corresponding relative deviation is greater than the normal deviation threshold, it is marked as an abnormal value; otherwise, it is marked as a normal value. Repeatedly retrieve all outliers from the initial incremental data in the data warehouse and remove them to obtain valid incremental data.

7. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 6, characterized in that, Preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size also includes the following sub-steps: Calculate the standard deviation YB of the effective incremental data, and set the partition size range to [DS, DL]; Set the neighbor radius to 0.8×YB and the minimum number of points to n. Perform density clustering on the effective incremental data to obtain multiple clusters. Calculate the proportion of data in each cluster to the total number of data in the effective incremental data, and record it as the cluster proportion. Clusters with a proportion greater than k6 within a cluster are designated as core clusters. If the proportion of any cluster is less than or equal to k6, then the k7 clusters with the largest proportion within a cluster are designated as core clusters. Here, k6 is the set proportion and k7 is the set number. Calculate the median of each core cluster, and then calculate the weighted average of the medians of all core clusters based on their proportion within the cluster. This average is denoted as the normal core increment VH.

8. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 7, characterized in that, Preprocessing the initial incremental data in the data warehouse and performing incremental size analysis to obtain the optimal partition size also includes the following sub-steps: Arrange the effective incremental data in chronological order and denote it as the second incremental sequence; let ES = "DS / VH", EL = ("DL / VH", 30), and use [ES, EL] as the range of values ​​for the merging multiple, where the merging multiple is an integer; Let any merging factor be E0. Starting from the beginning of the second increment sequence, calculate the sum of E0 consecutive unit-granularity data increments. If there are no E0 consecutive unit-granularity data increments, discard the result. Obtain the merged increment sequence corresponding to E0; VH×E0 is denoted as the corresponding candidate partition size SV. All corresponding merge increment sequences and candidate partition sizes are repeatedly obtained. For the candidate partition size SV corresponding to E0, calculate the standard deviation EB of the merged increment sequence corresponding to E0; Calculate (SV / DS) × (1-EB / YB), denoted as the combined earnings index of SV; Repeatedly obtain the merge yield index of all candidate partition sizes, and denote the candidate partition size with the largest merge yield index as the optimal partition size WS.

9. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 8, characterized in that, Adjusting the size of the time partition of the HUDI data lake according to the optimal partition size and obtaining the optimal bucket size involves the following sub-steps: Adjust the size of the finest time partition in the HUDI data lake to the optimal partition size; Set a candidate set of bucket sizes {128MB, 256MB}, calculate "WS / 128" and "WS / 256" respectively, and record them as the reasonable number of buckets F1 corresponding to 128MB and the reasonable number of buckets F2 corresponding to 256MB respectively; If both reasonable bucket numbers F1 and F2 are located in [1, DL / 256], then the bucket size corresponding to the smaller reasonable bucket number is taken as the optimal bucket size; if only one of F1 and F2 is located in [1, DL / 256], then the bucket size corresponding to the reasonable bucket number located in [1, DL / 256] is taken as the optimal bucket size.

10. The real-time data warehouse incremental synchronization method based on HUDI data lake according to claim 9, characterized in that, Adjusting the bucket size of the HUDI data lake based on the optimal bucketing information and performing real-time synchronization of data warehouse increments includes the following sub-steps: Adjust the bucket size of the time partition of the HUDI data lake to the optimal bucket size, set the adjustment period to T0, and periodically adjust the size of the time partition and bucket size of the HUDI data lake according to T0; Real-time data warehouse increments are captured, and corresponding data tables are created in HUDI according to the adjusted partition and bucket sizes. The newly added data is then placed in the corresponding positions in the data tables according to the rules, thus completing real-time data warehouse increment synchronization.

Images