Real-time storage optimization method based on block data

Abstract

The invention discloses a real-time storage optimization method based on block data, which relates to the technical field of big data processing and distributed storage, introduces a block data concept, realizes block-by-block writing, in-block ordering, block-level ordering and an intelligent flush trigger mechanism, and realizes real-time writing with high throughput, low delay and low memory occupation. The method specifically comprises the steps that block data is introduced into a data memory model to serve as a basic processing unit, multiple rows of data are aggregated into a block according to an aggregation key CKValue, row data in the block are sequenced according to a primary key PK, and write-in, namely aggregation is achieved; writing data, aggregating according to blocks, and uniformly sorting in the blocks; one block, namely multiple rows of data, generates an FRC file, and small files are reduced; exception processing: supporting block-level recovery to skip a failed block; the core problems of too high memory occupation, large CPU consumption, high writing delay and the like in a high-concurrency and high-throughput real-time writing scene can be solved, and the method is widely applied to scenes with extremely high requirements on data real-time performance and system stability.

Description

An Optimization Method for Real-Time Data Ingestion Based on Block Data Technical Field

[0001] This invention relates to the field of big data processing and distributed storage technology, and in particular to an optimization method for real-time data ingestion based on block data. Background Technology

[0002] In real-time data processing systems, HBase, as a distributed columnar storage system, is widely used in high-throughput, low-latency scenarios such as the Internet of Things (IoT), log analysis, and user behavior collection. Its core write component is MemStore, an ordered data structure residing in memory used to temporarily store data to be written.

[0003] The current mainstream MemStore write process is as follows: Data is written from the client to the RegionServer, using the rowkey as a unique identifier; before writing, it is necessary to check whether the rowkey exists and write it to the MemStore first; at the same time, WAL (Write-Ahead Long) is written to ensure that the data is not lost; when the MemStore reaches the threshold, a Flush operation is triggered to write the in-memory data to the HDFS storeFile.

[0004] In addition, to improve write performance, existing technologies also employ the following methods: batch writing: Flink / Spark Streaming aggregates streaming data into batch submissions; message queue buffering: Kafka acts as a buffer layer to decouple the data source from the database; compression and index optimization: StoreFile supports block compression, block indexing, and other mechanisms.

[0005] While the aforementioned techniques are effective in specific scenarios, and HBase MemStore offers advantages such as high throughput and low latency, HBase processes data on a row-by-row basis, meaning each row is written, sorted, and snapshotted independently. This "row-level" processing mode presents the following problems when facing high-frequency, batch write scenarios: 1) High sorting pressure: Every piece of data needs to be sorted, leading to rapid memory expansion under high concurrency, which can easily trigger JVMOOM (out-of-memory) errors.

[0006] 2) Small file overload: Frequent flushes generate a large number of small files, significantly increasing the pressure to merge.

[0007] 3) Mismatch in write granularity: In actual business operations, data is often processed in "blocks", and row-level processing is difficult to align with business semantics.

[0008] 4) High merging cost: There are many small files, making the merging task complex and affecting system stability.

[0009] Therefore, the FMDB real-time data memory model introduces the concept of "block data" on the basis of HBase MemStore, realizing a semantic upgrade from "row-level processing" to "block-level processing" to achieve a highly efficient real-time data ingestion architecture of "writing is aggregation, sorting is blockization, and flushing is blockization". Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide an optimization method for real-time data ingestion based on block data, which addresses the shortcomings of the prior art.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a block data real-time ingestion optimization method, which introduces the concept of block data, realizes block writing, ordered within the block, block-level sorting, and intelligent flush triggering mechanism, and realizes real-time writing with high throughput, low latency, and low memory usage; specifically includes the following steps: (1) Block data definition: In the data memory model, block data is introduced as the basic processing unit, and multiple rows of data are aggregated into a block according to the aggregation key CKValue. The row data within the block is sorted according to the primary key PK, so that writing is aggregation; (2) Block-level processing mechanism: data writing, block aggregation, and unified sorting within the block; data flush, one block, i.e., multiple rows of data, generates an FRC file to reduce small files; exception handling, supports block-level recovery to skip failed blocks.

[0012] As a further preferred embodiment of the block data real-time ingestion optimization method of the present invention, the data memory model includes Store and MemStore, wherein Store is a unified interface for the FMDB computing layer to access storage, and MemStore is the memory management unit of Store, used to manage data with block data as the basic unit.

[0013] As a further preferred embodiment of the block data real-time ingestion optimization method of the present invention, the data memory model also includes a Segment for providing real management of Row data; the Segment contains two derived classes, MutableSegment and ImmutableSegment; MutableSegment provides a write method, while ImmutableSegment does not; MutableSegment provides write capability as the active segment of MemStore; when MemStore creates an image, an ImmutableSegment is created by passing MutableSegment as a parameter, the memory layout remains unchanged, and the write capability is hidden.

[0014] As a further preferred embodiment of the block data real-time ingestion optimization method of the present invention, the Segment includes MemStoreLAB and RowSet; wherein, MemStoreLAB is responsible for requesting chunks from the memory pool and copying cell data into the chunks. In MSLAB, rows are stored in the ingestion order; RowSet is responsible for sorting the row data, realizing the ability to aggregate by CK and sort by PK.

[0015] As a further preferred embodiment of the block data real-time data ingestion optimization method of the present invention, the data writing specifically includes the following steps: The DBClient, i.e., the DBServer client, performs data cleaning on the real-time data, including verification and correction, and samples and records the data with verification errors. The verification includes: data partition verification, data type verification, and CKValue value verification; the correction includes: hiding fields, filling partition field values, correcting CKValue values, and avoiding the influence of prefix 0 and suffix 0 on ck queries; after the data verification is accurate, a partition is created based on the cleaned partition data; the DBClient Store information cache is obtained, hash(ckValue) is calculated, the data is grouped according to the Store range, and the cleaned data is converted into a byte[] structure; the grouped Stores are sent to the DBSlave, i.e., the DBServer slave, via RPC; after the DBSlave receives the data, it parses it and writes it to different stores, triggering flush according to the threshold.

[0016] As a further preferred embodiment of the block data real-time ingestion optimization method of the present invention, data flushing is performed by periodically checking the flush task in the flush request queue to perform a timely flush operation, specifically including the following steps: Flush triggering timing: The write service periodically obtains the list of stores from the region management and obtains the MemStoreSizing object of the corresponding store. This object contains the memory usage size of the store and the time information of the first piece of data written to the store; Memory size triggering: Short-cycle tasks periodically check whether the MemStoreSizing size corresponding to the data store has reached the threshold. If the threshold is reached, a flush call is made to flush the entire memstore; Time period triggering.

[0017] As a further preferred embodiment of the block data real-time ingestion optimization method of this invention, exception handling specifically includes the following steps: Data verification exceptions: When the client cleans data, it samples and records exception data including CK calculation errors, null value verification, original data column count and table schema size verification, and writes them into the ingestion error data directory; Client sending exceptions: When the client sends data to the DBSlave via RPC, and the receiving slave is in an abnormal state and cannot receive the RPC request normally, a delayed retry is performed. If it still fails after the configurable number of retries, the data is written to the failed data directory; Server writing exceptions: Table exceptions, partition exceptions, and exceptions not thrown to the client. These exceptions lead to For write failures, the number of failures and error types are recorded, and a client response is provided. For IO exceptions, the server retryes the process, and if the retry fails, it responds to the client with the error type and number of errors. If there is memory backlog in the memstore on the slave, a memmyIsTooBusy exception is thrown to the client, and the client performs data flooding without throwing it to the upper layer. If a bucket does not exist, a retry is required to avoid client caching due to bucket migration, splitting, or merging. d) Data reconciliation: DBClient receives data statistics, performs data volume statistics after data cleaning, and logs the data volume written to memory by DBSlave. DBSlave also performs data volume statistics for flushed data generated into files.

[0018] Compared with existing technologies, the present invention, employing the above technical solutions, has the following technical effects: The present invention provides a real-time data ingestion optimization method based on block data. Specifically, it references the HBase memory write buffer (MemStore) in the data memory model and introduces the concept of block data, moving from row-level processing to block-level processing. Lightweight sorting is achieved at the block level, reducing small file data and lowering merging pressure. This technology can solve core problems such as high memory consumption, high CPU consumption, and high write latency in high-concurrency, high-throughput real-time write scenarios, and is widely applicable to scenarios with extremely high requirements for data real-time performance and system stability. 1. Significantly reduces sorting pressure and improves system performance: In traditional methods, each row independently participates in global sorting, resulting in huge sorting overhead, especially with high CPU consumption in high-concurrency scenarios. The present invention performs unified sorting by PK and CK aggregation within the block, upgrading the sorting granularity from "row" to "block," achieving "order upon writing," transforming "high-concurrency sorting" into "batch centralized sorting," and achieving a performance leap. 2. Significantly reduces FRC small file data and optimizes storage structure: Traditional methods frequently flush, generating a large number of small files, leading to FRC... The number of files has increased dramatically; this invention achieves "Flush equals block" by having one CK block correspond to one FRC file during flushing; it fundamentally solves the problem of "small file proliferation" and improves system stability and maintainability; 3. It significantly reduces merging pressure and improves system throughput. Traditional methods result in a large number of small files, leading to heavy compaction tasks, long merging times, and impacting write performance; this invention reduces the amount of compaction tasks by more than 70% due to a significant reduction in the number of FRC files; it achieves "write-merge" decoupling and ensures smooth system operation in high-concurrency scenarios. Attached Figure Description

[0019] Figure 1 is a flowchart of the data writing process to menstore according to the present invention; Figure 2 is a schematic diagram of the data storage model according to the present invention; Figure 3 is a flowchart of the data writing process according to the present invention; Figure 4 is a flowchart of the flush process of the memory data according to the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings: The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0021] This invention proposes a real-time data ingestion optimization technology based on block data. Its core idea is to introduce the concept of "block data" to achieve block-based writing, ordered writing within blocks, block-level sorting, and intelligent flush triggering mechanism, thereby realizing a real-time writing system with high throughput, low latency, and low memory consumption.

[0022] Block data definition: Multiple rows of data are aggregated into a block according to CKValue. Rows within the block are sorted by PK. CK is used for aggregation, and aggregation occurs when data is actually written.

[0023] Block-level processing mechanism, as shown in Table 1: Table 1 Processing Stage Traditional HBase (row-level) This Invention (block-level) Write Each row is written independently, sorted row by row. Aggregated by block, unified sorting within the block. Sorting Each row participates in global sorting, high pressure. Sorting within the block reduces sorting volume by 80%+. Flush Multiple rows are written in a distributed manner, generating small files. One block (multiple rows of data) generates one FRC file, reducing small file recovery. Recovery Only supports row-level recovery. Supports block-level recovery, can skip failed blocks. surface

[0024] (1) Data Memory Model: Store serves as the unified interface for the FMDB computing layer to access storage. MemStore is the memory management unit of Store. There is only one MemStore instance in a Store. Block data is used as the basic data unit. Multiple rows are aggregated into a block. Rows within a block are sorted by PK and clustered by CK. They are ordered as they are written. All Rows are written to the chunk as a continuous byte stream. UnsafeRow encoding is supported. Block-level snapshots, block-level sorting, and block-level lifecycle management are implemented in the MemStore layer to achieve integrated optimization of writing into blocks, snapshots into blocks, and flushing into blocks. MemStore: The memory management unit in Store. There is only one MemStore instance in a Store. When real-time data is entered into the database, it is first written to MemStore. When the data exceeds a specified threshold, it is flushed into an FRC file. FRC: A file format optimized and modified based on ORC, called FRC. CKValue: Multiple rows of data are aggregated into a block according to CKValue. The blocks are sorted by PK (primary key). Row: A piece of data.

[0025] Store is the unified entry point for the computing layer to access the storage layer in Fiberhome Massive Database (FMDB). It provides interfaces for writing and reading data and primarily handles interactions with the file system. The relationships between Stores are managed by the computing layer, and there are no dependencies or coupling between Stores. Fiberhome Massive Database (FMDB) refers to Fiberhome's next-generation massive database.

[0026] The data writing process to the MemStore is shown in Figure 1 below: MemStore is the memory management unit within the Store. There is only one MemStore instance in a Store, shielding implementation details from external access. It provides two interfaces, upsert and add, for writing data and supports snapshot() for generating snapshots.

[0027] A MemStore manages two Segments: one for writing and the other for serialization caching. The MemStore does not actually manage the data; it only handles the state management of the two Segments.

[0028] Store's add / upsert calls the MemStore interface to write data. When the DBServer detects insufficient memory and needs to flush, it calls the Store's flush interface. In the Store's flush process, the MemStore Snapshot interface is first called to generate an image, then the data in the MemStoreSnapshot is read and serialized to an FRC file. Once a Snapshot serialization is complete, the entire memory is released.

[0029] MemStore's image generation involves converting a MutableSegment into an ImmutableSegment as a Snapshot, while simultaneously creating a new MutableSegment as the active segment for writing data.

[0030] Segment has two derived classes: MutableSegment and ImmutableSegment. MutableSegment provides write methods, while ImmutableSegment does not.

[0031] MutableSegment provides write capability as the active segment of MemStore. When creating an image in MemStore, MutableSegment is simply used as a parameter to create an ImmutableSegment; the memory layout remains unchanged, only the write capability is disabled.

[0032] Segment provides real-time management of Row data, and the data storage model is shown in Figure 2.

[0033] A Segment contains two main objects: MemStoreLAB (memstore local allocation buffer): It's responsible for allocating chunks from the memory pool and copying cell data into the chunks. In MSLAB, rows are stored in the order they were added. The copy process follows this logic: if the row size is greater than the threshold (256K), the original cell is returned (cell3 in Figure 2); otherwise, the cell is copied to MSLAB, and a new cell is created, with the memory pointer pointing to the offset in the chunk returned. The copy process is as follows: if the remaining memory size of the current chunk is greater than the row size, the row is written to the chunk; otherwise, a chunk is allocated for copying. RowSet: Responsible for sorting the row data, requiring the ability to aggregate by CK and sort by PK. Segments provide data reading through SegmentScanner; both the query and flush processes read data through this mechanism. To control memory release, a refCnt is introduced in MSLAB. The counter is incremented by 1 when a SegmentScanner is created and decremented by 1 when it is closed. MSLAB releases memory when the counter reaches 0. This mechanism ensures that multiple read processes can read concurrently while also releasing memory promptly.

[0034] MemStoreLAB: Provides pooled memory management capabilities for MemStore, mainly implementing cell writing and chunk management capabilities.

[0035] Chunk management is achieved through a ConcurrentSkipListSet. <intger>Record all requested chunks. Based on the application counter mechanism (NIO implementation), the entire requested chunk is released by the last caller using `close`.

[0036] MemPool: Chunk: A contiguous unit of memory for management purposes.

[0037] Data is managed through ByteBuffer, which is implemented using both offheap and onheap.

[0038] It has several data attributes: id (int), nextFreeOffset (int), chunkType, and fromPool. In both the pool and MSLAB, chunks are associated using this ID.

[0039] The `alloc(int size)` method is provided. This method essentially checks if the remaining capacity of the chunk is less than `size`. If it is, it returns the current offset and adds `size` to that offset as the new offset. Otherwise, it returns -1. When the value is -1, MSLAB will allocate a new chunk for writing data.

[0040] A chunk is actually a memory management unit that is written sequentially. When modifying a cell added to a pool, its actual length cannot increase, otherwise it will lead to data errors.

[0041] ChunkCreator: The unit for memory allocation and pool management.

[0042] When requesting a chunk, priority is given to obtaining a chunk from the pool. If the pool is empty, a chunk is requested from the JVM.

[0043] During release, if the total size of the pool is less than the threshold, the chunk is added to the pool. Otherwise, no action is taken, and the chunk is reclaimed by garbage collection.

[0044] A Pool is a BlockingQueue that manages multiple chunks of equal size.

[0045] (2) Data writing, the flowchart is shown in Figure 3: DBClient (DBServer client) cleans the real-time data (verifies and corrects) and samples and records the data with verification errors. The main verifications include: data partition verification, data type verification, and CKValue value verification; the corrections include: hidden fields, filling partition field values, correcting CKValue values, and avoiding the influence of prefix 0 and suffix 0 on ck query.

[0046] After the data is verified to be accurate, partitions are created based on the cleaned partition data.

[0047] Retrieve DBClient Store information cache, perform hash(ckValue) calculation, group the data according to the Store range, and convert the cleaned data into byte[] structure.

[0048] The grouped Stores will send data via DBSlave (DBServer slave) RPC.

[0049] After receiving the data, DBSlave parses it and writes it to different stores, triggering a flush according to the threshold.

[0050] DBServer is the core computing engine of FMDB, implemented based on an M / S architecture.

[0051] (3) Data flush: The flush process of memory data is shown in Figure 4. The flush operation is performed at the right time by periodically checking the flush task in the flush request queue.

[0052] Flush trigger timing: The write service periodically retrieves the list of stores from the region management and obtains the MemStoreSizing object of the corresponding store. This object contains the memory usage size of the store and the time information of the first piece of data written to the store.

[0053] Memory size trigger: Short-cycle tasks periodically check whether the MemStoreSizing size corresponding to the data store has reached the threshold. If the threshold is reached, a flush call is made to flush the entire memstore.

[0054] Before each batch of writes, a global check is performed, and a low watermark and a high watermark are set. The global low watermark is the default value of slave memory: 60%, and the high watermark is the default value of slave memory: 80%.

[0055] When the memory level reaches a high level, select the memory to be flushed to the low watermark: 60% according to the store size order.

[0056] The time period triggers a check to see if the first piece of data written to the store has expired. The default time is 5 minutes. If it exceeds 5 minutes, a flush is performed.

[0057] (4) Exception handling: Data verification exception: When the client cleans the data, it samples and records the abnormal data (ck calculation error, null value verification, original data column number and table shema size verification) and writes them into the database error data directory.

[0058] Client sending exception: When the client sends data to DBSlave via RPC, the receiving slave is in an abnormal state and cannot receive the RPC request normally. In this case, a delayed retry is performed. The number of retries can be configured (default 1). If it still fails after the retries, the data is written to the failure data directory.

[0059] Server-side write exceptions: table exceptions, partition exceptions, and exceptions not thrown to the client. If a write fails due to such exceptions, the number of failed entries and the error type will be recorded, and a client response will be sent.

[0060] In case of IO exceptions, the server will retry the request. If the retry fails, the server will respond to the client with the error type and the number of errors.

[0061] If the Slave experiences memory backlog in the memstore, it throws a memeryIsTooBusy exception to the client. The client then performs data flooding and does not throw the exception to the upper layer.

[0062] Buckets do not have any issues requiring retries to avoid client caching issues caused by bucket migration, splitting, or merging. A Bucket is a logical unit for organizing and managing data according to certain encoding rules; it's a directory on HDFS that manages data shards.

[0063] Data reconciliation: DBClient receives data statistics, performs data volume statistics after data cleaning, and logs the data.

[0064] DBSlave memory write data volume statistics.

[0065] Statistics on the amount of data generated by flushing data in DBSlave.

[0066] The real-time FRC file import process is as follows: 1. Create a database table; 2. Execute the import command and create a topic in Kafka to push data; 3. View the FRC files generated by HDFS and count the imported data. There are a total of 70,000 rows of data, but only 4 FRC files.

[0067] This invention significantly reduces sorting pressure and improves system performance: Traditional methods involve each row independently participating in global sorting, resulting in huge sorting overhead, especially with high CPU usage in high-concurrency scenarios; This invention performs unified sorting by PK and CK aggregation within blocks, upgrading the sorting granularity from "row" to "block," achieving "order upon writing," transforming "high-concurrency sorting" into "batch centralized sorting," and achieving a performance leap; it greatly reduces FRC small file data and optimizes the storage structure: Traditional methods frequently flush, generating a large number of small files, leading to a surge in the number of FRC files; This invention, during flushing, corresponds to one FRC file per CK block, achieving "flush as blockization"; it fundamentally solves the problem of "small file proliferation," improving system stability and maintainability; it significantly reduces merging pressure and improves system throughput: Traditional methods involve a large number of small files, resulting in heavy compaction tasks, long merging times, and impacting write performance; This invention, due to the significant reduction in the number of FRC files, reduces the amount of compaction tasks by more than 70%; it achieves "write-merge" decoupling, ensuring smooth system operation in high-concurrency scenarios.

[0068] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0069] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / intger>

Claims

1. An optimization method for real-time data ingestion based on block data, characterized in that: The concept of block data is introduced to realize block writing, block ordering, block-level sorting, and intelligent flush triggering mechanism to achieve high throughput, low latency, and low memory usage in real-time writing; specifically, the following steps are included: (1) Block data definition: In the data memory model, block data is introduced as the basic processing unit. Multiple rows of data are aggregated into a block according to the aggregation key CKValue. The row data in the block is sorted according to the primary key PK to realize writing and aggregation; (2) Block-level processing mechanism: Data writing, block aggregation, and unified sorting within the block; Data flush, a block of multiple rows of data generates an FRC file to reduce small files; Exception handling, supporting block-level recovery to skip failed blocks.

2. The method for optimizing real-time data ingestion based on block data according to claim 1, characterized in that: The data memory model includes Store and MemStore. Store is the unified interface for the FMDB computing layer to access storage, and MemStore is the memory management unit of Store, used to manage data in blocks of data.

3. The method for optimizing real-time data ingestion based on block data according to claim 2, characterized in that: The data memory model also includes Segments, which provide actual management of Row data. Segments contain two derived classes: MutableSegment and ImmutableSegment. MutableSegment provides write methods, while ImmutableSegment does not. MutableSegment provides write capabilities as the active segment of MemStore. When creating an image in MemStore, an ImmutableSegment is created by passing MutableSegment as a parameter, with the memory layout unchanged and write capabilities disabled.

4. The method for optimizing real-time data ingestion based on block data according to claim 3, characterized in that: The Segment contains MemStoreLAB and RowSet; MemStoreLAB is responsible for requesting chunks from the memory pool and copying cell data into the chunks. In MemStoreLAB, rows are stored in the order they were entered into the database. RowSet is responsible for sorting the row data, enabling aggregation by CK and sorting by PK.

5. The method for optimizing real-time data ingestion based on block data according to claim 1, characterized in that: Data writing specifically includes the following steps: The DBClient (DBServer client) cleans the real-time data, including verification and correction, and samples and records the data with verification errors. Verification includes: data partition verification, data type verification, and CKValue value verification. Correction includes: hiding fields, filling partition field values, correcting CKValue values, and avoiding the impact of prefix and suffix 0 on CK queries. After the data verification is accurate, partitions are created based on the cleaned partition data. The DBClient Store information cache is obtained, hash(ckValue) is calculated, and the data is grouped according to the Store range. At the same time, the cleaned data is converted into a byte[] structure. The grouped Stores are sent to the DBSlave (DBServer slave) via RPC. After receiving the data, the DBSlave parses it and writes it to different stores, triggering flush according to the threshold.

6. The method for optimizing real-time data ingestion based on block data according to claim 1, characterized in that, Data flushing involves periodically checking flush tasks in the flush request queue and performing flush operations accordingly. Specifically, it includes the following steps: Flush triggering timing: The write service periodically retrieves a list of stores from the region management system and obtains the corresponding MemStoreSizing object. This object contains the store's memory usage and the time information of the first data entry written to that store; Memory size triggering: Short-cycle tasks periodically check whether the MemStoreSizing size corresponding to the data store has reached a threshold. If it does, a flush call is made to flush the entire memstore; Time period triggering.

7. The method for optimizing real-time data ingestion based on block data according to claim 1, characterized in that, The exception handling process includes the following steps: Data validation exceptions: When the client cleans data, it samples and records exceptions such as CK calculation errors, null value validation, and validation of the original data column count and table schema size, and writes them to the database error data directory; Client sending exceptions: When the client sends data to the DBSlave via RPC, and the receiving slave is in an abnormal state and cannot receive the RPC request normally, a delayed retry is performed. If the retry still fails after a configurable number of times, the data is written to the failed data directory; Server writing exceptions: For write failures caused by table exceptions, partition exceptions, or exceptions not being thrown to the client, the number of failed entries and the error type are recorded, and a client response is provided. For IO exceptions, the server will retry on the server side. If the retry fails, it will respond to the client with the error type and number of errors. If there is a memory backlog in the memstore in the slave, it will throw a memeryIsTooBusy exception to the client. The client will then perform data flooding and will not throw it to the upper layer. If there is no Bucket, it needs to be retried to avoid client caching due to bucket migration, splitting, or merging. d) Data reconciliation: DBClient receives data statistics, performs data volume statistics after data cleaning, and logs the data; DBSlave writes data volume statistics to memory; DBSlave flushes data and generates files with data volume statistics.

Images