ES-based data import optimization method and system

Abstract

The invention relates to an ES-based data import optimization method and system. The method comprises the following steps: acquiring source data needing to be imported into an ES, and initializing an ES fragment group and a Hadoop cluster; performing fragmentation preprocessing on the source data by utilizing a Hadoop cluster to obtain a source data fragmentation group; according to the source data fragment groups, generating an index file conforming to an ES bottom-layer format, wherein each source data fragment group corresponds to one fragment in an ES cluster; compressing the index file by using a Zstand-ard compression algorithm to generate a compressed data block; distributing the compressed data block to a target node in an ES cluster according to a dynamic load fragmentation strategy; and the target node decompresses the compressed data block, and imports an index file obtained after decompression into the rear end of the current ES index file to complete import.

Description

Technical Field

[0001] This application relates to the field of data import, and mainly to a data import optimization method and system based on ES. Background Art

[0002] As internet business data volumes continue to grow exponentially, ES, as a core distributed search engine, is tasked with efficiently storing and retrieval petabytes of data. Data import, a primary step in ES applications, directly impacts system responsiveness and user experience. However, current mainstream import solutions rely primarily on ES APIs or bulk APIs. While these solutions meet basic requirements, they exhibit significant shortcomings when dealing with massive amounts of data.

[0003] The current ES data import method is difficult to meet real-time requirements. The single-machine thread pool and synchronous IO mechanism will lead to limited throughput. When importing data into the ES cluster, it will greatly increase the CPU, memory, network bandwidth and other resources of the ES server, leading to serious resource utilization problems. At the same time, the current ES data import method lacks scalability. Due to the rigid sharding strategy, the load is uneven, and hot data nodes are prone to performance shortcomings. There is a lack of breakpoint resumption and automatic rebalancing mechanisms, and data compression efficiency is low.

[0004] For example, the Chinese invention patent with publication number "CN113590703A" discloses an "ES data import method, device, electronic device and readable storage medium", which specifically discloses "performing sharding processing on the imported data to obtain a first number of sharded data; allocating the first number of sharded data to a first number of computing nodes in the computing cluster, and controlling the first number of computing nodes to generate index data for the corresponding sharded data; creating directory information for the ES cluster, downloading each index data to the corresponding ES node in the ES cluster based on the directory information, and merging the index data belonging to the same block in each ES node to obtain the target data corresponding to each ES node". However, this method downloads the index data generated by the computing node to each node of the ES cluster through the network, which will cause network transmission delay and bandwidth consumption, affecting the overall import efficiency; in addition, the sharding allocation of the computing nodes in this method does not take into account the real-time load of the ES nodes, which will cause some nodes to be overloaded or uneven resource utilization, affecting the overall performance of the cluster. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present application provides a data import optimization method and system based on ES.

[0006] The technical solution of this application is as follows:

[0007] In one aspect, the present invention proposes a data import optimization method based on ES, the method comprising:

[0008] Obtain the source data to be imported into ES and initialize the ES shard group and Hadoop cluster; use the Hadoop cluster to pre-shard the source data to obtain source data shard groups; generate index files that conform to the ES underlying format based on the source data shard groups, where each source data shard group corresponds to a shard in the ES cluster;

[0009] Compress the index file using the Zstandard compression algorithm to generate compressed data blocks; distribute the compressed data blocks to the target nodes in the ES cluster according to the dynamic load sharding strategy;

[0010] The target node decompresses the compressed data block, and imports the decompressed index file into the backend of the current ES index file to complete the import.

[0011] Preferably, the method further comprises performing data cleaning on the source data, wherein the data cleaning comprises processing missing values, outliers and standardizing the data format.

[0012] Preferably, the source data is pre-processed by sharding using a Hadoop cluster, specifically:

[0013] Extract the data primary key of the source data and calculate the hash value of the data primary key; assign the source data to the ES shard group according to the hash value of the data primary key, and obtain the source data shard group and the index file that conforms to the underlying format of ES;

[0014] Each source data shard group generates an index file that conforms to the underlying ES format.

[0015] Preferably, the index file is compressed using the Zstandard compression algorithm to generate compressed data blocks, specifically:

[0016] Dividing the index file into data blocks of a preset size specifically comprises rounding up a byte ratio of the index file to each data block to obtain the number of data blocks obtained after the division;

[0017] Determine the bytes of the index file and the preset byte size of each data block to obtain the byte size of the current data block;

[0018] The current data block is compressed using the Zstandard compression algorithm to obtain a compressed data block.

[0019] Preferably, the compressed data blocks are distributed to the target nodes in the ES cluster according to the dynamic load sharding strategy, specifically:

[0020] Calculate the load rate of nodes in the ES cluster using the formula:

[0021]

[0022] Where R k represents the load rate of the kth node; L k represents the comprehensive load of the kth node; L k,m Cap represents the load of the kth node for the mth indicator; k represents the processing capacity of the kth node; w m represents the weight of the mth indicator; M represents the number of indicators; m represents the index value of the mth indicator; k represents the index value of the kth node;

[0023] The exponential smoothing method is used to update the weight of the indicator, which can be expressed as follows:

[0024]

[0025] Where, represents the weight of the mth indicator at the t+1th iteration; α represents the preset smoothing coefficient; t represents the index value of the tth iteration;

[0026] The node with the smallest load rate is selected as the target node, the compressed data block is distributed to the target node in the ES cluster, and the load of the current target node is updated.

[0027] Preferably, the decompressed index file is imported into the backend of the current ES index file. Specifically, if the number of existing segments of the current ES index file exceeds a preset threshold, the file is imported and the number of existing segments is updated.

[0028] Preferably, the method also includes deploying a lightweight monitoring agent to collect the load status of each node in the ES cluster in real time, and predicting the load trend of the node in combination with a sliding time window algorithm; if the predicted result of the current node load trend is higher than a preset threshold, the processing capacity of the current node is improved, and the remaining tasks on the current node are migrated to a node with a load rate lower than the preset threshold.

[0029] On the other hand, the present invention also proposes a data import optimization system based on ES, which includes a data acquisition module, a data preprocessing module and a data import module, wherein:

[0030] The data acquisition module is used to obtain the source data that needs to be imported into ES and initialize the ES shard group and Hadoop cluster; transmit the source data, ES shard group and Hadoop cluster to the data preprocessing module;

[0031] The data preprocessing module is used to perform shard preprocessing on the source data using the Hadoop cluster to obtain source data shard groups and index files that conform to the underlying format of ES, wherein each source data shard group corresponds to a shard in the ES cluster;

[0032] Compress the index file using the Zstandard compression algorithm to generate compressed data blocks; distribute the compressed data blocks to the target nodes in the ES cluster according to the dynamic load sharding strategy;

[0033] The data import module is used for the target node to decompress the compressed data block, import the index file obtained after decompression into the backend of the current ES index file, and complete the import.

[0034] On the other hand, the present invention also proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an ES-based data import optimization method as described in any embodiment of the present invention is implemented.

[0035] On the other hand, the present invention further proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an ES-based data import optimization method as described in any embodiment of the present invention.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1) The present invention provides an ES-based data import optimization method and system. This system utilizes a Hadoop cluster to build a distributed sharding preprocessing system. Source data is intelligently allocated to ES sharding groups based on a data primary key hash modulo algorithm, and independent index files are generated for each sharding group. Leveraging the Hadoop cluster's distributed computing framework, data sharding, which previously required sequential processing on a single machine, can now be performed simultaneously by multiple nodes, improving data sharding speed and overall data processing throughput.

[0038] 2) The present invention provides an ES-based data import optimization method and system, which uses the Zstandard high-efficiency compression algorithm to compress index files. This algorithm, relying on dictionary matching and context modeling technology, achieves a high compression ratio while maintaining a decompression speed of GB / second. Compared with traditional compression algorithms, it reduces network bandwidth usage by more than 40% and disk storage space by 60%, thereby improving data transmission efficiency and storage resource utilization, reducing bandwidth usage and storage costs, and improving disk space utilization.

[0039] 3) This invention provides an ES-based data import optimization method and system, utilizing a dynamic load sharding strategy to address the performance bottlenecks associated with traditional static resource allocation. By monitoring node metrics in real time and conducting comprehensive analysis, the system provides a comprehensive understanding of each node's load and dynamically adjusts resource allocation. When assigning tasks, the system prioritizes the node with the lowest current load. This prevents some nodes from being constantly highly loaded while others remain idle. This enhances real-time responsiveness, improves node utilization, and avoids the performance bottlenecks associated with traditional static resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0042] The present invention provides the following technical solution: a data import optimization method and system based on ES.

[0043] Example 1

[0044] See Figure 1 This embodiment provides a data import optimization method based on ES, and the specific steps include:

[0045] S1. Obtain the source data to be imported into ES and initialize the ES shard group and Hadoop cluster.

[0046] The method further includes performing data cleaning on the source data, wherein the data cleaning includes processing missing values, outliers and standardizing the data format;

[0047] S2. Use the Hadoop cluster to pre-process the source data into shards to obtain source data shard groups; generate index files that conform to the underlying ES format based on the source data shard groups, where each source data shard group corresponds to a shard in the ES cluster;

[0048] Extract the data primary key of the source data and calculate the hash value of the data primary key; assign the source data to the ES shard group according to the hash value of the data primary key, and obtain the source data shard group and the index file that conforms to the underlying format of ES, which can be expressed as follows:

[0049] gi =V i mod N;

[0050] V i =Hash(key i );

[0051] S={S1,...,S g ,...,S N};

[0052] Where g i Represents the index of the i-th source data shard group; V i Represents the hash value of the primary key of the i-th source data; key i represents the primary key of the i-th source data; N represents the number of preset source data shard groups; mod represents the modulus function; Hash() represents the hash function; i represents the index value of the i-th source data; S g Indicates the g-th source data shard group; g indicates the index value of the g-th source data shard group;

[0053] Each source data shard group generates an index file that conforms to the underlying ES format and is represented as Among them Seg g Indicates the segment file of the g-th source data shard group, F f Represents the f-th index file, and f represents the index value of the f-th index file;

[0054] S3. Compress the index file using the Zstandard compression algorithm to generate a compressed data block;

[0055] The index file is divided into data blocks of a preset size by rounding up the byte ratio of the index file to each data block to obtain the number of data blocks obtained after the division, which is expressed as follows:

[0056]

[0057] Where n represents the number of data blocks obtained after partitioning; |F| represents the byte size of the index file; B represents the preset byte size of each data block; represents the ceiling function;

[0058] Determine the bytes of the index file and the preset byte size of each data block to obtain the byte size of the current data block;

[0059] Among them, if |F|>B, then |F j |=B,j=0,...,n-1,where j represents the index value of the jth data block; otherwise, |F n-1 |=|F|-(n-1)B;

[0060] Use the Zstandard compression algorithm to compress the current data block to obtain the compressed data block, which is expressed as follows:

[0061] C j =Zstandard(F j );

[0062] Where C j Indicates the jth data block after compression; Zstandard() indicates the Zstandard compression algorithm;

[0063] S4. Allocate the compressed data block to the target node in the ES cluster according to the dynamic load sharding strategy;

[0064] Calculate the load rate of nodes in the ES cluster using the formula:

[0065]

[0066] Where R k represents the load rate of the kth node; L k represents the comprehensive load of the kth node; L k,m Cap represents the load of the kth node for the mth indicator; k represents the processing capacity of the kth node; w m represents the weight of the mth indicator; M represents the number of indicators; m represents the index value of the mth indicator; k represents the index value of the kth node;

[0067] In this embodiment, the indicators include CPU usage, memory usage, and network bandwidth;

[0068] The exponential smoothing method is used to update the weight of the indicator, which can be expressed as follows:

[0069]

[0070] Where, represents the weight of the mth indicator at the t+1th iteration; α represents the preset smoothing coefficient; t represents the index value of the tth iteration;

[0071] Select the node with the smallest load rate as the target node, which can be expressed as:

[0072]

[0073] Where, TN represents the target node; argmin() represents the function to obtain the minimum load rate;

[0074] Distribute the compressed data block to the target node in the ES cluster and update the load of the current target node, which can be expressed as:

[0075]

[0076] Where, represents the load of the target node at the t+1th iteration; β represents the load update coefficient; T represents the processing data; C j Indicates the byte size of the jth data block after compression;

[0077] S5. The target node decompresses the compressed data block and imports the decompressed index file into the backend of the current ES index file. Specifically, if the number of existing segments of the current ES index file exceeds a preset threshold, the file is imported and the number of existing segments is updated. The size of the existing segments after merging is expressed as Among them Seg new Indicates the existing segment, O z The zth merged segment, that is, the index file obtained after the zth decompression, z represents the index value of the zth merged segment, and N represents the preset number of source data shard groups. The import is completed.

[0078] S6. The method also includes deploying a lightweight monitoring agent to collect the load status of each node in the ES cluster in real time, and predicting the load trend of the node in combination with a sliding time window algorithm; if the predicted result of the current node load trend is higher than a preset threshold, the processing capacity of the current node is improved, and the remaining tasks on the current node are migrated to a node with a load rate lower than the preset threshold.

[0079] Example 2

[0080] This embodiment provides an ES-based data import optimization system, which includes a data acquisition module, a data preprocessing module, and a data import module, wherein:

[0081] The data acquisition module is used to obtain the source data that needs to be imported into ES and initialize the ES shard group and Hadoop cluster; transmit the source data, ES shard group and Hadoop cluster to the data preprocessing module;

[0082] The data preprocessing module is used to perform shard preprocessing on the source data using the Hadoop cluster to obtain source data shard groups and index files that conform to the underlying format of ES, wherein each source data shard group corresponds to a shard in the ES cluster;

[0083] Compress the index file using the Zstandard compression algorithm to generate compressed data blocks; distribute the compressed data blocks to the target nodes in the ES cluster according to the dynamic load sharding strategy;

[0084] The data import module is used for the target node to decompress the compressed data block, import the index file obtained after decompression into the backend of the current ES index file, and complete the import.

[0085] Example 3

[0086] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, an ES-based data import optimization method as described in any embodiment of the present invention is implemented.

[0087] Example 4

[0088] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the ES-based data import optimization method as described in any embodiment of the present invention is implemented.

[0089] It is worth noting that the system, electronic device and computer-readable storage medium described in the present invention are all based on the same principles as the method described in Example 1, and will not be repeated here.

[0090] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data import optimization method based on ES, characterized in that: The method comprises: Obtain the source data to be imported into ES and initialize the ES shard group and Hadoop cluster; use the Hadoop cluster to pre-shard the source data to obtain source data shard groups; generate index files that conform to the ES underlying format based on the source data shard groups, where each source data shard group corresponds to a shard in the ES cluster; Compress the index file using the Zstandard compression algorithm to generate compressed data blocks; distribute the compressed data blocks to the target nodes in the ES cluster according to the dynamic load sharding strategy; The target node decompresses the compressed data block, and imports the decompressed index file into the backend of the current ES index file to complete the import.

2. The data import optimization method based on ES according to claim 1 is characterized in that: The method further includes performing data cleaning on the source data, wherein the data cleaning includes processing missing values, outliers and standardizing data formats.

3. The data import optimization method based on ES according to claim 1 is characterized in that: The source data is pre-processed by sharding using the Hadoop cluster, specifically: Extract the data primary key of the source data and calculate the hash value of the data primary key; assign the source data to the ES shard group according to the hash value of the data primary key, and obtain the source data shard group and the index file that conforms to the underlying format of ES; Each source data shard group generates an index file that conforms to the underlying ES format.

4. The data import optimization method based on ES according to claim 1 is characterized in that: The index file is compressed using the Zstandard compression algorithm to generate compressed data blocks, specifically: Dividing the index file into data blocks of a preset size specifically comprises rounding up a byte ratio of the index file to each data block to obtain the number of data blocks obtained after the division; Determine the bytes of the index file and the preset byte size of each data block to obtain the byte size of the current data block; The current data block is compressed using the Zstandard compression algorithm to obtain a compressed data block.

5. The data import optimization method based on ES according to claim 4 is characterized in that: The compressed data blocks are distributed to the target nodes in the ES cluster according to the dynamic load sharding strategy, specifically: Calculate the load rate of nodes in the ES cluster using the formula: Where R k represents the load rate of the kth node; L k represents the comprehensive load of the kth node; L k,m Cap represents the load of the kth node for the mth indicator; k represents the processing capacity of the kth node; w m represents the weight of the mth indicator; M represents the number of indicators; m represents the index value of the mth indicator; k represents the index value of the kth node; The exponential smoothing method is used to update the weight of the indicator, which can be expressed as follows: Where, represents the weight of the mth indicator at the t+1th iteration; α represents the preset smoothing coefficient; t represents the index value of the tth iteration; The node with the smallest load rate is selected as the target node, the compressed data block is distributed to the target node in the ES cluster, and the load of the current target node is updated.

6. The data import optimization method based on ES according to claim 1 is characterized in that: The decompressed index file is imported into the backend of the current ES index file. Specifically, if the number of existing segments of the current ES index file exceeds a preset threshold, the file is imported and the number of existing segments is updated.

7. The data import optimization method based on ES according to claim 1 is characterized in that: The method also includes deploying a lightweight monitoring agent to collect the load status of each node in the ES cluster in real time, and using a sliding time window algorithm to predict the load trend of the node; If the predicted result of the current node load trend is higher than the preset threshold, the processing capacity of the current node is improved, and the remaining tasks on the current node are migrated to nodes with a load rate lower than the preset threshold.

8. A data import optimization system based on ES, characterized in that: The system includes a data acquisition module, a data preprocessing module and a data import module, wherein: The data acquisition module is used to obtain the source data that needs to be imported into ES and initialize the ES shard group and Hadoop cluster; transmit the source data, ES shard group and Hadoop cluster to the data preprocessing module; The data preprocessing module is used to perform shard preprocessing on the source data using the Hadoop cluster to obtain source data shard groups; generate index files that conform to the ES underlying format based on the source data shard groups, wherein each source data shard group corresponds to a shard in the ES cluster; Compress the index file using the Zstandard compression algorithm to generate compressed data blocks; distribute the compressed data blocks to the target nodes in the ES cluster according to the dynamic load sharding strategy; The data import module is used for the target node to decompress the compressed data block, import the index file obtained after decompression into the backend of the current ES index file, and complete the import.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the ES-based data import optimization method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, an ES-based data import optimization method as described in any one of claims 1 to 7 is implemented.

Images