OLAP engine hot-cold separation optimization method using object storage as cold storage

Abstract

The present invention discloses a hot-cold separation optimization method for an OLAP engine using object storage as cold storage, including: a control center is responsible for the full process control of hot and cold, the control center periodically scans partitions, and when a partition reaches a cooling condition, the automatic merge function of all copies of the partition is turned off, and a cooling command is issued by the control center; a suitable cooling point is found, one of the copies is selected to execute the cooling process, and the data before the cooling point is merged. The merging process separates pure data and metadata, and the separated files are migrated to a remote object storage. Sharing remote cold data: The control center notifies other copies to download and load metadata files from the remote end, and after cooling is completed, the optimized automatic merge mechanism is started. The present invention also optimizes queries for hot and cold scenarios. The present invention has low storage costs, shares remote copies, solves the existing copy redundancy problem, can control the performance degradation of storage media, supports the situation where cold partitions continue to write data, and improves query efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular relates to an OLAP engine cold-hot separation optimization method using object storage as cold storage. Background Art

[0002] In the Internet industry, data is becoming more and more important, and data-driven business is increasingly recognized. However, the amount of data generated every day is increasing, resulting in increasing storage and computing costs, and the query performance requirements for data are also increasing. There are more and more demands for real-time / quasi-real-time reports, and the amount of data stored is increasing, and the cost is increasing accordingly. At present, the storage cost is memory > flash memory (SSD) > mechanical disk (HDD) > cloud disk, but the performance is the opposite.

[0003] In the Internet industry, data is usually time-sensitive, and its importance generally decreases over time. Data within the last seven days is of high importance and has a high query frequency. Data queries after one month are rare and have decreased importance, but data cannot be completely discarded. Historical data queries are still required for historical audits and replays. In response to different values ​​and query frequencies, as well as the requirements for query response speed, the market has proposed tiered data processing, where high-priority data is generally stored in SSDs, low-priority data is stored in HHDs, and cool data is stored in cloud disks.

[0004] There are two main methods for hot and cold separation in existing databases. Solution 1: Use network disk mapping. For example, Clickhouse can use JuiceFS mapping to map cloud storage such as S3 to local disks. In this way, reading and writing remote cloud disks are processed like local disks. Figure 1 As shown, the local cloud disk directory is mapped to a low-cost disk. The schematic diagram takes a partition as an example. Usually a table consists of one or more partitions. In order to ensure the reliability of data, a partition is saved in multiple copies, and the data between the copies is the same.

[0005] The main disadvantage of this solution is that one local copy of data corresponds to one copy of data in the cloud. To ensure data reliability, the cloud generally has three copies. If the local database also uses three copies, then for one real data, there are 3*3=9 copies of data. Therefore, the consumption is too large, and the mapping method and query performance are completely dependent on the mapping system, and the performance is greatly affected by the mapping system.

[0006] Solution 2: Actively write data to the remote end

[0007] Compared with the cloud disk mapping system, actively controlling cold data to the remote end can do more customized optimization. Figure 2As shown in the figure, when the replica data cools down, the system actively sends the data file to the remote storage medium. When querying, the remote data will be reloaded to the local and cached. Generally, common database partitions no longer support data writing after cooling down. The data query and pull unit is the file. The cooling data storage method is based on replicas, which are isolated from each other and do not interfere with each other.

[0008] The existing active cooling cloud disk solution has the following shortcomings:

[0009] 1) Data is cooled by partitions, but most systems do not support updating or adding data to partitions after cooling. Systems that support updates generally have a lot of IO problems due to the compaction process.

[0010] 2) Cold data is based on files, so no matter what kind of query is performed, the entire file will be downloaded, which has a great impact on the query performance of cold data; after the query, the entire file will be cached on the local disk to speed up repeated queries, but for cold data that is not frequently queried, the role of the cache will be much smaller, and the resource overhead will be greater.

[0011] 3) Common database cooling solutions generally use a copy to manage cold data separately. Similar to the mapping method, there is too much redundancy in the copy, which cannot achieve real resource saving.

[0012] The main difference between Solution 1 and Solution 2 is that Solution 1 relies heavily on the file mapping system, does not do any control, and is simple to implement; Solution 2 actively controls the entire process and can control performance and storage by itself, but the cost is that it is slightly more difficult to implement. The two solutions are essentially the same, so the problems they bring are similar. Summary of the invention

[0013] The present invention discloses an OLAP engine cold-hot separation optimization method using object storage as cold storage, comprising the following steps:

[0014] S1 Select cooling point: The control center is responsible for the whole process control of hot and cold. The control center regularly scans the partitions and selects the partitions that meet the trigger conditions. When the partitions meet the cooling conditions, the control center issues a cooling command.

[0015] For a partition that meets the trigger condition, the automatic merging process of all copies of the partition is stopped first, and then the cooling point is found. The data before the cooling point participates in the cooling, and the data after the cooling point does not participate in this cooling;

[0016] S2 performs cooling: selects one of the replicas and performs a complete small file merging process on the data before the cooling point. During the merging, the pure data and metadata files are separated and the separated data are migrated to a remote low-cost storage medium. The cooling process does not affect the writing of new data. If there is both hot and cold data or only one of the hot and cold data, the replicas share a remote data copy. The entire cooling process does not block the partition from appending data.

[0017] After the file is migrated to the remote end, the control center notifies other replicas of the partition to download the metadata file from the remote object storage. After the download is complete, the control center notifies the cooling data to be enabled, loads the local metadata into the memory, saves the data remotely, and then starts the optimized basic merge process of the S3 step. The replicas share a copy of the remote data, and the entire process does not block the partition data writing.

[0018] S3 optimized basic merge: basic merge is performed on local new data, and remote data is not involved. A full merge is performed during the next cooling period. The full merge involves merging both remote and local data.

[0019] S4 query optimization: Store indexes and data separately to reduce query performance degradation caused by different media. In point query scenarios, the block offset of a specific file can be quickly located through metadata. Only a small block file needs to be downloaded, reducing data download.

[0020] Optimize query plans. For SQL statements with Limit conditions, prioritize querying local hot data to improve performance. Incorporate cold data query IO factors into the cost model to generate a better query plan.

[0021] Furthermore, the step S2 includes:

[0022] When a partition meets the cooling requirement, the control center first notifies the data node where the partition replica is located to stop the merging function of the partition. After receiving the request, the data node stops the incremental merging and basic merging.

[0023] The control center asks the data node where the partition replica is located to report the maximum version number of the replica. The control center uses the maximum version number of the replica as the cooling point.

[0024] The control center selects a data node where one of the partition replicas is located, and re-merges the data before the cooling point; migrates the re-merged metadata files and pure data files to low-cost storage media, and deletes the local data files;

[0025] The control center notifies other replicas of the partition to download the partitioned metadata information from the remote end;

[0026] After the complete download, the control center notifies the activation of cooling data, loads the local metadata into the memory, and saves the data file remotely, guided by the metadata index;

[0027] The control center notifies the data node where the partition replica is located to delete the original data before the cooling point;

[0028] The Control Center notifies the optimized base merge process that restarts the S3 step.

[0029] Furthermore, the re-merging will merge all the files, but the generated new files will separate the pure data and metadata files, and the metadata includes the block index and secondary index of the data.

[0030] Furthermore, in step S3, after the data cools down, an optimized merge strategy is used when performing a basic merge on the newly written data. This strategy adjusts the scope of Delete so that Delete is only effective for remote cold data. The optimized basic merge does not pull remote data, thereby reducing IO resource consumption.

[0031] Furthermore, the metadata includes at least one of a secondary index, a primary key, and a block index.

[0032] Furthermore, the query optimization in step S4 further includes:

[0033] For JION scenarios, introduce IO cost into the cost model to find the most reasonable way;

[0034] Reduce the download of cold data;

[0035] Introduce block cache to locally cache high-frequency query blocks.

[0036] Furthermore, the remote object storage includes at least one of OBS, OSS, and cloud disk.

[0037] Furthermore, the remote storage includes at least one of network disk, S3, and HDFS cloud storage.

[0038] The beneficial effects of the present invention are as follows:

[0039] The present invention provides an OLAP hot-cold separation optimization solution based on object storage (cloud storage), which saves resources through low-cost storage media. Its actual storage cost is about 1 / 10 of HHD and 1 / 20 of SSD. In terms of query efficiency, some queries are not affected or even faster, and point queries can respond in about 500 milliseconds. The average query performance of TPC-H cold data is about 1.5 times that of local hot data storage (HHD), and can be within 1.3 times after eliminating SQL queries within 1 second.

[0040] The present invention shares remote copies and solves the existing copy redundancy problem.

[0041] The present invention controls the performance degradation of the storage medium by separating the data file and the metadata (including the data index and the secondary index), which is particularly obvious in the scene of point query or large amount of data filtering.

[0042] After the data is cooled, the present invention provides a new compaction solution, which solves the problem that a large amount of remote data needs to be pulled for each cold data base compaction, thereby supporting the situation where cold partitions continue to write data.

[0043] The present invention optimizes commonly used queries by optimizing SQL, reduces the query process for cold partitions, and improves query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the hot-cold separation solution for traditional single-partition multi-copy disk mapping;

[0045] Figure 2 Schematic diagram of the traditional hot-cold separation solution for active writing to remote storage;

[0046] Figure 3 A schematic diagram of the relationship between cooling and merging of the present invention;

[0047] Figure 4 Data cooling execution flow chart of the present invention;

[0048] Figure 5 Schematic diagram of the merging solution for coexistence of cold and hot data. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0050] In the e-commerce industry, data tables are generally partitioned based on time. The main queries and additions of data are focused on the most recent time periods. The amount of data query and addition of data in long-term partitions will be relatively small. Data tables will be partitioned according to time, and historical data will be cooled down regularly and then migrated.

[0051] The present invention is based on the OLAP engine. OLAP adopts a table-based storage method. The table is composed of partitions. For reliability, a partition generally has multiple copies. The data between the copies is exactly the same. When a single copy fails, it can be automatically repaired by copying the copy. OLAP is divided into a control node (MASTER) and a data node (Data Node). The control node manages the metadata at the table level and controls the execution of the process. The control node manages the entire process of hot and cold separation, and the data node replicates and executes the specific process.

[0052] In the OLAP engine, data is written to the memory first and then flushed to the disk. After being written to the disk, the data file cannot be directly modified or some records in it cannot be directly deleted. The engine updates by writing new record files. When querying, the old files are filtered. Data deletion is not executed immediately, but only marked for deletion. Then, the marked deleted data is filtered during the query process to achieve the deletion effect. The actual deletion process is in the basic merge.

[0053] During the Base-Compaction process, the new files generated by the Base-Compaction do not contain the deleted data, so the delete records can be truly deleted after the Base-Compaction.

[0054] Merges are divided into base merges and incremental merges. The main purpose of merges is to reduce the number of files under the partition. The more files there are, the worse the query efficiency. Base merges (base-compaction) will load the full amount of data in the partition copy. The full amount of data is involved to achieve true deletion; incremental merges only merge a small number of newly written files. Incremental merges generally have a high frequency, and base merges have a slightly lower frequency. Basic merges require the participation of all data, which means that if the data is cooled to the remote end, if a basic merge is to be performed, the remote cold data needs to be loaded into the database and then participate in the merge. Therefore, in general designs, cold data partitions do not support data writing and updating. The cooling scheme of the present invention does not affect the data writing and updating of cold partitions. The entire cooling separation scheme is as follows:

[0055] 1. Select cooling point:

[0056] The control center is responsible for the whole process control of cooling and heating. The control center regularly scans the partitions and selects the partitions that meet the trigger conditions. When the partitions meet the cooling conditions, the control center issues cooling commands.

[0057] For the partition that meets the trigger, the automatic merging process of all copies of the partition will be stopped first, and then the cooling point will be found. The data before the cooling point will participate in the cooling, and the data after the cooling point will not participate in this cooling. The trigger condition is that the time of the partition reaches the threshold, such as one partition per day, it can be cooled after 30 days (threshold), and the cooling can be triggered again after 60 days. Each time the cooling is performed, it will be judged whether there is new data written in the cycle. If no new data is written, no cooling will be performed in this cycle. In some embodiments, non-partition scenarios can also be supported. The difference between non-partition and time-based partitions lies in the selection of cooling points. For time-based partition scenarios, the maximum version in the partition is used as the cooling point. If it is a non-partition scenario, data selection needs to be based on the writing time of the data, and the operability is relatively low, and such data generally has a small amount of data.

[0058] Data cooling is also based on partitions. Partitions are the basic units of data storage. Partitions form tables. A partition has multiple copies, and the data between copies is exactly the same. The compaction inside the copy is executed separately. In the cooling process of the present invention, data writing is not prohibited within the partition, but before cooling the data, it is necessary to stop the system's automatic compaction process and then find the cooling point. The data before the cooling point participates in the cooling, and the data after cooling does not participate in this cooling. If the compaction process is not stopped, it will lead to the following: Figure 3 The structure, such as Figure 3 As shown, there are 3 replicas in total, and the merging between replicas is controlled automatically. At the beginning, each replica has 3 files. Replica 1 has been merged once, and replicas 2 and 3 have not been merged. If Compaction is not turned off and a cooling point is selected at this time, then replica 2 may trigger a merge after new data is written. When replica 2 generates a new file, the cooling point will be inside the merged file, which is very inconvenient for the migration of cold data. Therefore, the present invention first prohibits merging, and then finds the cooling point. When the cold data is migrated to the remote end, the merging function is enabled.

[0059] 2. Perform cooling:

[0060] Select one of the replicas in step 1 and perform a complete small file merging process on the data before the cooling point. During the merging process, the data file and the metadata file will be separated. The merged data and metadata will be uploaded to the low-cost remote object storage.

[0061] The control center (Master) is responsible for the whole process control of hot and cold. The Master will scan the partitions regularly. When the partitions reach the cooling condition, the control center will issue a cooling command. After finding a suitable cooling point, the data before the cooling point is migrated to a low-cost remote object storage, and the data after the cooling point is retained locally, waiting for the next cooling. Figure 4 .

[0062] 1) When the partition meets the cooling requirement, the control center will first notify the Data Node where the partition copy is located to stop the compaction function of the partition. After receiving the request, the Data Node will stop the incremental merge and basic merge.

[0063] 2) The control center asks the Data Node where the partition replica is located to report the maximum version number of the replica. The control center uses the maximum version number of the replica as the cooling point.

[0064] 3) The control center selects the Data Node where one of the partition replicas is located to perform Recompaction based on the cooling point. Recompaction is the same as Base Compaction, which will merge all files, but the generated new files will separate the data and metadata files. The metadata includes basic information such as the block index and secondary index of the data. The data offset position can be accurately located through the secondary index and block index information.

[0065] 4) Migrate the Recompaction metadata files and data files to remote storage media with lower storage capacity, such as OBS, OSS, cloud disk, etc., and delete local data files. Remote storage also includes network disk, S3, HDFS cloud storage, etc.

[0066] 5) The control center notifies other replicas of the partition to download the metadata information after Repartition from the remote end.

[0067] 6) After the download is complete, the control center notifies the cooling data to be enabled and loads the local metadata into the memory. The data files are stored remotely and are guided by the metadata index.

[0068] 7) The control center notifies the Data Node where the partition replica is located to delete the original data at the cooling point.

[0069] 8) The control center notifies to restart a new automatic merge process, i.e. step 3.

[0070] After the new merge scheme is enabled, the cooling process is completed. Data can be written normally during and after the cooling process. New data will be written locally and treated as hot data. Cold data is the same data used by different copies. The controllability of cold data is guaranteed by the cloud storage itself, which truly saves resources. The object storage cost price used in the present invention is about 45 yuan per terabyte (commercial large-scale price, data reliability, and copy price already included), and the local cloud disk is 140 yuan per terabyte, and 3 copies are required to ensure data reliability. Therefore, the cost of cold data is about 1 / 10 of that of the local cloud disk (45 / 140 / 3).

[0071] 3. Optimize the basic merging process

[0072] The main purpose of merging is to reduce small files and actually clear deleted data. Merging is divided into basic merging and incremental merging. In order to clear junk data, basic merging must involve all files. Normally, each basic merge must pull remote data, which consumes too much IO. By optimizing basic merging, only local new data is merged, and the remote is not involved. The next time the cooling is done, a full merge (both remote and local) is performed.

[0073] When data in a partition is uploaded to remote storage, the incremental merging of the new data is no different from before. The main change is in the basic merging, because the basic merging requires constantly pulling the full amount of data from the remote end, and the cost of pulling the full amount of data is relatively high.

[0074] The basic merge has two main functions: 1. Reduce the number of files; 2. OLAP uses marked deletion, and the basic merge can achieve true deletion. Only then can the marked statement be cleared. During the cooling period, the remote data has been recompacted once, and the files have been reduced to the minimum. Generally, no further reduction is required, so it does not need to participate in subsequent basic merges. Here we mainly solve the deletion action of the basic merge. The specific solution for the remote end not to participate in the local Base Compaction is as follows Figure 5 .

[0075] like Figure 5 After the data cools down, data record 1 is newly written, and then the Delete statement is executed, and then record 2 is written. The Delete statement only effectively filters data record 1 and cold data. When Base-Compaction is performed on hot data, all hot data participates in the merger, but cold data does not participate.

[0076] 4. Query optimization

[0077] After the data is cooled, query problems will also be involved. In the cooling solution of the present invention, the index and data are stored separately. The purpose is to reduce the query performance degradation caused by the difference in media. In the point query scenario, the present invention quickly locates the block offset of a specific file through metadata such as secondary indexes, primary keys, and block indexes. Only a small block of files needs to be downloaded, and the query performance can reach within 500 seconds. The performance is higher for non-IO query statements, such as count(*). In the TPC-H test, the average query performance of cold data is about 1.5 times that of the local hot data storage (HHD), and it can be achieved within 1.3 times after eliminating SQL queries within 1s. In addition, other types of SQL have also been optimized, such as SQL with Limit conditions, which prioritizes hot data queries. For JION scenarios, IO costs are introduced into the cost model to find the most reasonable way. Reduce the download of cold data. In addition, block cache is introduced to locally cache high-frequency query blocks, which has a smaller granularity and is more flexible than file cache.

[0078] The beneficial effects of the present invention are as follows:

[0079] The present invention provides an OLAP hot-cold separation optimization solution based on object storage (cloud storage), which saves resources through low-cost storage media. Its actual storage cost is about 1 / 10 of HHD and 1 / 20 of SSD. In terms of query efficiency, some queries are not affected or even faster, and point queries can respond in about 500 milliseconds. The average query performance of TPC-H cold data is about 1.5 times that of local hot data storage (HHD), and can be within 1.3 times after eliminating SQL queries within 1 second.

[0080] The present invention shares remote copies and solves the existing copy redundancy problem.

[0081] The present invention controls the performance degradation of the storage medium by separating the data file and the metadata (including the data index and the secondary index), which is particularly obvious in the scene of point query or large amount of data filtering.

[0082] After the data is cooled, the present invention provides a new compaction solution, which solves the problem that a large amount of remote data needs to be pulled for each cold data base compaction, thereby supporting the situation where cold partitions continue to write data.

[0083] The present invention optimizes commonly used queries by optimizing SQL, reduces the query process for cold partitions, and improves query efficiency.

[0084] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0085] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0086] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0087] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cold-hot separation optimization method for an OLAP engine using object storage as cold storage, characterized in that: The following steps are involved: S1 Select cooling point: The control center is responsible for the whole process control of hot and cold. The control center regularly scans the partitions and selects the partitions that meet the trigger conditions. When the partitions meet the cooling conditions, the control center issues a cooling command. For a partition that meets the trigger condition, the automatic merging process of all copies of the partition is stopped first, and then the cooling point is found. The data before the cooling point participates in the cooling, and the data after the cooling point does not participate in this cooling; S2 performs cooling: select one of the replicas and merge the data before the cooling point into small files. During the merging process, the pure data and metadata files are separated and the separated data are migrated to a remote low-cost storage medium. The cooling process does not affect the writing of new data. When both hot and cold data exist or only one of the hot and cold data exists, the replicas share a copy of remote data, and the entire cooling process does not block the partition from appending data; After the file is migrated to the remote end, the control center notifies other replicas of the partition to download the metadata file from the remote object storage; After the download is complete, the control center notifies the cooling data to be enabled, loads the local metadata into the memory, saves the data remotely, and then starts the optimized basic merge process of the S3 step; The replicas share a copy of remote data, and the entire process does not block partition data writing; S3 optimized basic merge: basic merge is performed on local new data, and remote data is not involved. A full merge is performed during the next cooling period. The full merge involves merging both remote and local data. S4 query optimization: Store indexes and data separately to reduce query performance degradation caused by different media. In point query scenarios, the block offset of a specific file can be quickly located through metadata. Only a small block file needs to be downloaded, reducing data download. Optimize query plans. For SQL statements with Limit conditions, prioritize querying local hot data to improve performance. Incorporate cold data query IO factors into the cost model to generate a better query plan.

2. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 1 is characterized in that: The step S2 comprises: When a partition meets the cooling requirement, the control center first notifies the data node where the partition replica is located to stop the merging function of the partition. After receiving the request, the data node stops the incremental merging and basic merging. The control center asks the data node where the partition replica is located to report the maximum version number of the replica. The control center uses the maximum version number of the replica as the cooling point. The control center selects a data node where one of the partition replicas is located, and re-merges the data before the cooling point; migrates the re-merged metadata files and pure data files to low-cost storage media, and deletes the local data files; The control center notifies other replicas of the partition to download the partitioned metadata information from the remote end; After the complete download, the control center notifies the cooling data to be enabled, loads the local metadata into the memory, and saves the data file remotely, guided by the metadata index; The control center notifies the data node where the partition replica is located to delete the original data before the cooling point; The Control Center notifies the optimized base merge process that restarts the S3 step.

3. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 2 is characterized in that: The re-merging merges all the files, but the generated new files separate the pure data and metadata files, and the metadata includes the block index and secondary index of the data.

4. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 1 is characterized in that: In step S3, after the data cools down, an optimized merge strategy is used when performing a basic merge on the newly written data. This strategy adjusts the scope of Delete so that Delete only takes effect on remote cold data. The optimized basic merge does not pull remote data, reducing IO resource consumption.

5. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 1 is characterized in that: The metadata includes at least one of the secondary index, primary key, and block index.

6. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 1, characterized in that: The query optimization in step S4 further includes: For JION scenarios, introduce IO cost into the cost model to find the most reasonable way; Reduce the download of cold data; Introduce block cache to locally cache high-frequency query blocks.

7. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 2 is characterized in that: The remote object storage includes at least one of OBS, OSS, and cloud disk.

8. The cold-hot separation optimization method of the OLAP engine using object storage as cold storage according to claim 2 is characterized in that: The remote storage also includes at least one of network disk, S3, and HDFS cloud storage.

Images