Large-scale relational data storage and paging query method based on HBase

Abstract

The invention discloses an HBase-based large-scale relational data storage and paging query method, which comprises the following steps of: realizing bidirectional traversal according to node type priority ranking by adopting a composite row key single-row storage relation edge; a sequence index + Bloom filter is used for eliminating offset holes, and the empty Get rate is reduced to 0.3%; through 7-byte compression Token + 60s short transaction renewal, the RPC bandwidth is reduced by 50%; re-hashing the online hot spots to improve the peak value writing by 40%; mVCC visibility is locked by a millisecond-level T0 snapshot, and multi-level cache is matched, so that average 28ms of deep paging of ten millions of pages and 45ms of P99 are realized; the method supports gigabit-edge PB-level storage, is 100% compatible with HBase ecology, and is suitable for scenes such as social contact, atlas and risk control.

Description

Technical Field

[0001] This invention relates to the field of big data storage and query technology, and in particular to a large-scale relational data storage structure and pagination query method based on HBase, which is applicable to application scenarios that require the storage and querying of massive relational data, such as social networks, knowledge graphs, recommendation systems, and financial risk control. Background Technology

[0002] With the rapid development of the Internet and IoT technologies, the scale and complexity of relational data extracted and mined from massive datasets are growing exponentially. Traditional relational databases maintain relationships using foreign keys and indexes, but in distributed environments, they exhibit problems such as difficulty in scaling, weak concurrency capabilities, and exponentially declining performance of deep pagination. NoSQL databases such as HBase, due to their excellent horizontal scalability and high throughput, are gradually becoming the preferred solution for processing massive amounts of data.

[0003] HBase, a distributed column-oriented database based on HDFS, boasts advantages such as horizontal scalability, high throughput, and low-latency random read / write, making it widely used to address storage needs for tables with hundreds of billions of rows. However, HBase natively supports key-value or wide-column models and lacks direct support for relational semantics. Existing solutions typically employ the following methods: 1. Adjacency list: Each edge is stored as a separate row. Querying all neighbors of a node requires a full table scan, which greatly amplifies I / O. 2. Bidirectional redundancy: In order to support two queries, "A's friends" and "who is A's friend", two rows are stored for each edge, doubling the amount of writing and making maintenance more complex; 3. Graph database add-ons: Introducing dedicated systems such as Neo4j and JanusGraph, although semantically rich, the cluster size is limited by the sharding strategy, and the horizontal scaling capability is not as good as HBase; 4. Hybrid architecture: HBase is used to store attributes online, and Spark is used to calculate relationships offline. This results in long system chains, poor timeliness, and high operation and maintenance costs.

[0004] In terms of pagination queries, HBase only provides PageFilter based on LIMIT, and cannot use OFFSET like SQL. This means that when performing "deep pagination", all previous data must be scanned sequentially, and the latency increases linearly with the page number. At the same time, in scenarios such as Region splitting and concurrent writes, the same pagination request may return duplicate or missing data, lacking consistency guarantees.

[0005] In the prior art, an invention patent with publication number CN103617232A discloses a pagination query method for HBase tables. To address the inefficiency of traditional HBase pagination queries (such as displaying data page by page or jumping to a specific page), especially the linear decrease in query efficiency as the data volume increases, an index table is introduced: a sequential number (serial number) is established for each row of data in the HBase main table (Data table), and the mapping relationship between "serial number → RowKey" is stored in a separate index table. Atomic incrementing serial numbers: the serial numbers in the index table are atomically incremented, ensuring that the serial numbers are unique and continuous when data is inserted. During the pagination query process, based on the page number requested by the user and the number of rows per page, the required data serial number range is calculated. All RowKeys corresponding to this serial number range are retrieved from the index table at once, and then the actual data is queried in batches from the main table based on these RowKeys. This patent supports arbitrary page jumps: because the index table provides a global serial number mapping, users can jump directly from page 1 to any page N without scanning page by page.

[0006] The aforementioned patent CN103617232A implements pagination queries using RowKey-cursor pagination, but in practical applications, data drift occurs during deep pagination. This is because each write request (Put / Delete) is assigned a sequentially increasing write sequence number (sequenceId) on the server side. Each Scan also receives a read sequence number (readPoint) equal to the largest completed sequenceId the moment it opens. For the same row, HBase retains multiple versions, each with its own sequenceId; only versions with sequenceId ≤ readPoint are visible to the current Scan. Therefore, a "single Scan" itself does not read newly inserted data from other versions; this is the basic snapshot capability of MVCC. User pagination is not a "single Scan to the end," but rather: The first RPC: Scan[startRow=S0,stopRow=E,limit=20]; Second RPC: Scan[startRow=S1=last line of the first scan + minimum offset,stopRow=E (upper bound of this scan),limit=20]; Therefore, the following problems exist between the two RPCs: 1. Before new data is inserted into the current page (RowKey falls between [S0, S1]); 2. Or, older data may have been deleted / modified, making the version unavailable; 3. Alternatively, a region may split or merge, causing the region boundaries to shift.

[0007] Because the readpoint for the second scan is recalculated, the snapshot it sees is no longer the global state at the moment of the first scan. Therefore: the same logical record may appear repeatedly (visible in both old and new snapshots); or it may be missed forever (visible the first time, deleted / overwritten by a new version the second time); in the worst case, the total number of results fluctuates when paging continuously, leading the client to believe "the scan is complete" when there is still data remaining. In other words, current technology relying solely on RowKey-cursor paging will experience drift and missed counts during deep paging. Summary of the Invention

[0008] A brief overview of embodiments of the invention is provided below to provide a basic understanding of certain aspects of the invention. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0009] This invention aims to address the problems of low efficiency and poor pagination performance in existing HBase relational data storage technologies. It provides a method for HBase relational data storage and pagination querying, overcoming the shortcomings of existing technologies through methods such as normalized RowKey, column family optimization, MVCC snapshot pagination, hotspot distribution, and multi-level caching. The method achieves the following effects: bidirectional traversal can be supported with only one row stored per edge, eliminating redundant data; millisecond-level response time is maintained even for deep pagination at the level of tens of millions of pages; data insertion or deletion during pagination does not affect the current session result, ensuring consistency; it supports hundreds of billions of edges and petabytes of storage, with linear cluster scalability; and it is compatible with the existing HBase ecosystem, requiring no additional graph database or computing engine.

[0010] Specifically, this application provides a method for large-scale relational data storage and paginated query based on HBase, including the following steps: a) A composite row key structure is used to standardize and store relational data. The composite row key includes: starting node type, starting node identifier, target node type, target node identifier, relation edge type, and relation generation timestamp. b) Sort the row keys according to the preset node type priority, with the higher priority row key always placed to the left of the row key to achieve bidirectional traversal without redundant storage; the node type priority and sorting result are applied to both the main table write and subsequent sequential index construction. c) Construct information column families to store basic relationship attributes, and metadata column families to store system metadata, and support dynamic column qualifier expansion; the information column families are used to be returned to the client from the scan results; the metadata column families are used to store the last modification time when renewing short transactions, in order to determine whether a snapshot needs to be regenerated. d) Before the first query, the client generates a global snapshot version number T0. In subsequent pagination requests, visibility is locked by setting a time range (0, T0), and the consistency of pagination results is ensured by combining the MVCC mechanism. The whitelist key for setting the time range (0, T0) is a composite row key. T0 is written to both the cache key and the new token, so that when the cache is hit, the T0 truncation value is compared first. If the minute part is different, it is forcibly invalidated, ensuring that the cache view is consistent with the snapshot. The unit of T0 is milliseconds, which refers to the millisecond-level timestamp obtained by System.currentTimeMillis() when the client first initiates a pagination request. The cache key includes a user identifier, page number, and a minute-level truncation value for T0 to ensure consistency between the cache view and the snapshot view; the minute-level truncation value is aligned with the short transaction renewal period. e) Construct pagination queries based on sequential index row key range scans, setting start and end row keys to limit the scan range, using the last row key returned in the previous scan as the starting position of the next scan, and using the original row key contained therein to batch look up the main table, combined with page filtering to limit the number of pages returned, so that the deep pagination latency is stabilized at the millisecond level; the boundary parameters of the start and end row keys, as well as the last row key, are all obtained from the sequential index row key; f) Use hash prefix or string reversal strategies for high-frequency node identifiers to avoid regional hotspots; directly replace the starting node identifier or target node identifier field in step a with the node identifier after hash prefix or string reversal, and apply it to the sharding prefix of the subsequent sequential index to ensure that the main table and the index table use the same hashing rules.

[0011] Furthermore, it also includes the following steps: g) Utilize block caching, bucket caching, and business-local caching to build a multi-level caching system, and adopt asynchronous prefetching to reduce query latency; The block cache and bucket cache are used to accelerate sequential index scanning; the business local cache is linked to the minimum value of the page filtering parameter in step e, and asynchronous prefetching is triggered when the remaining entries in the business local cache are less than the minimum value of the page filtering parameter.

[0012] To address the issue of data drift and omissions during deep pagination, this application employs an MVCC mechanism to ensure data consistency during pagination queries. To achieve this, step d assigns a global timestamp to the entire pagination cycle: before the client's first request, System.currentTimeMillis() is used to record it as T0; this timestamp is then distributed along with the page token as the business snapshot version number. Then, step e forces each scan to only view data ≤ T0. Because HBase's TimeRange filtering performs a whitelisting after the MVCC version visibility check, any new versions written after T0 (even if the sequenceId has been committed) will be completely blocked. Meanwhile, subsequent page turns reuse the same T0, and the client brings T0 back to the server. The second, third and so on page turn requests continue to setTimeRange(0,T0). The entire page turn session always sees the same snapshot. Region splitting, merging, new insertion, and deletion have no effect. Therefore, there will be no duplication, no missing counts, and the total number of records is fixed. In other words, by using TimeRange to extend the "MVCC snapshot" from a single scan to the entire page turn session, drifting data is completely locked out.

[0013] In practical applications, under special circumstances involving long transactions and deep pagination, deep pagination requires sequentially scanning a large number of rows to locate the "starting key of the Nth page"; the scanning process does not immediately release read versions to prevent rollback; if the user's pagination session is prolonged (background export, BI report), the transaction ID is not closed, and HBase cannot prune all old versions before T0, resulting in: an increase in the number of StoreFiles and a surge in disk usage; the read path needs to traverse more version chains, leading to a double increase in CPU and IO, resulting in an exponential degradation of "the further you scroll, the slower it gets." Therefore, the biggest contradiction that the MVCC mechanism brings to pagination is that "fixed snapshots" and "dynamic offsets" cannot be achieved simultaneously: to fix the snapshot, T0 must be locked → potentially leading to gaps, version inflation, and cache mismatch; to offset in real time, a new snapshot needs to be replaced, which may result in duplicate / missing data issues. Existing solutions to such problems usually require multiple compensation mechanisms such as "client-side deduplication + cursor renewal + snapshot renewal + boundary verification" to balance consistency and performance in big data deep pagination scenarios. To overcome the above problems, this application also includes the following process: h) Limit the snapshot validity period to no more than 1 minute. When the pagination session expires, regenerate the snapshot and return a new token in T0+1ms to achieve short transaction renewal. The T0 in the new token is written back to the set time range parameter in step d and synchronously updated to the cache key, enabling short transaction renewal without rebuilding the scanned data; the cache key is forcibly invalidated when the minutes are crossed to prevent the return of expired snapshots; i) Write a sequential index row asynchronously for each relation edge. The sequential index row key consists of a sharding prefix, a monotonically increasing sequence number and the original row key. Page scanning is performed only on the sequential index. The sharding prefix and the hash prefix in step f share the same hash function; the monotonically increasing sequence number is used for continuous positioning of the starting row key of the sequential index scan to eliminate offset gaps caused by intermediate insertions; and the original row key list parsed from the sequential index is used as input (Table.batchGet(List<original row key>)) to perform batch reading of the main table and obtain the information column family data in step c.

[0014] Furthermore, step j) is added: a 3-byte segmented Bloom filter is appended to the end of the sequential index row key, and local filtering is performed before batch Get to reduce empty Gets; in the scenario of 120 billion edges, the empty Get rate is reduced from 12% to 0.3%, and the total latency is reduced by another 15%.

[0015] Furthermore, it also includes the online hotspot rehashing step k): The RegionServer counts QPS in real time. When the QPS of a single Region is greater than the threshold, it triggers an online "prefix bit + 1" split without stopping writing; the QPS of the hotspot Region is reduced from 120,000 to 30,000, and the peak write speed of the cluster is increased by 40%.

[0016] Furthermore, in step h, data versions prior to the time range (0, T0-60000ms (i.e., 1 minute)) are allowed to be physically reclaimed to reduce the risk of version backlog.

[0017] Furthermore, the monotonically increasing sequence number in step i is generated by the Snowflake algorithm, and the sequence number is lexicographically consecutive within the row key.

[0018] Furthermore, the cache key is in the format userId:pageNo:{T0 / / 60000} and expires automatically across minutes.

[0019] Furthermore, a 1-2 byte variable-length Gamma-encoded weight value is appended after the relation edge type of the composite row key, forming "Start Node Type | Start Node Identifier | Target Node Type | Target Node Identifier | Relation Edge Type | gamma(weight) | Relation Generation Timestamp"; where gamma(weight) is the variable-length encoding of the weight value. The gamma(weight) weight value is a one-byte variable-length encoding with a value range of 0-65535 (unsigned); its encoded length is 1-2 bytes (1B for ≤127; 2B for ≥128); its placement is immediately after the edge type, ensuring that edges of the same type and with similar weights are physically continuous. Range scanning can be completed by directly defining byte boundaries using StartRow / StopRow, eliminating the need for client-side filtering.

[0020] When querying the weight range [1000, 2000], only the byte boundaries of gamma(1000) and gamma(2000) need to be calculated, and StartRow / StopRow can be set directly without Filter; in the scenario of 120 billion edges, the average latency of Scan in the weight range is reduced from 800ms to 50ms, and the storage volume increases by <2%.

[0021] As a specific implementation scheme, this application uses a 42-bit millisecond timestamp + 22-bit cursor offset to form 56-bit data, which is only 7 bytes after Base64 encoding; during lease renewal, only delta=1ms is transmitted. Compared with plaintext T0 + lastRowKey, the token size is reduced by 50%, and the lease renewal RPC QPS is increased by 3 times.

[0022] The specific implementation of the 7-byte compressed token is as follows: it uses a 42-bit millisecond timestamp + a 22-bit cursor offset to form 56 bits of data, which is only 7 bytes after Base64 encoding; when renewing the lease, only delta=1ms is transmitted.

[0023] Furthermore, the method of this invention supports hundreds of billions of relational edges and petabyte-level storage, with an average latency of less than 30ms for deep paging.

[0024] Furthermore, the node type priority is a preset order, such as USER < PRODUCT < CATEGORY < SHOP, to ensure consistent RowKey sorting.

[0025] This invention implements a relational data storage and pagination query method for HBase, including: using composite row keys to store relational edges in a single row, and sorting by node type priority to achieve bidirectional traversal; introducing variable weight encoding to reduce the query latency of weight intervals to 50ms; constructing a sequential index + Bloom filter to eliminate offset holes and reduce the empty Get rate to 0.3%; using 7-byte compressed tokens + short transaction renewal to reduce RPC bandwidth by 50%; online hotspot rehashing to improve peak write speed by 40%; and locking the T0 snapshot through MVCC + TimeRange, combined with multi-level caching, to achieve an average deep pagination time of 28ms for tens of millions of pages and 45ms for P99. This invention supports petabyte-level storage with hundreds of billions of edges, is 100% compatible with the HBase ecosystem, and is suitable for scenarios such as social networking, graph analysis, and risk control. Compared with existing technologies, it achieves the following beneficial effects: 1. A single edge only needs to be stored in one row to support bidirectional traversal, saving more than 40% in storage; 2. Deep pagination at the level of tens of millions of pages still maintains millisecond-level response (average 28ms, P99 45ms); 3. Data insertion or deletion during page turning does not affect the current session result, ensuring consistency; 4. Supports hundreds of billions of edges and petabytes of storage; the cluster can be linearly scaled up to 200 RegionServers. 5. 100% compatible with the existing HBase ecosystem, and can be directly integrated with Phoenix, Spark, and Flink for analysis. Attached Figure Description

[0026] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts. These drawings, together with the following detailed description, are incorporated in and form part of this specification, and are used to further illustrate preferred embodiments of the invention and explain the principles and advantages of the invention. In the drawings: Figure 1 This is a flowchart of the pagination query method of this application. Detailed Implementation

[0027] Embodiments of the present invention will now be described with reference to the accompanying drawings. Elements and features described in one drawing or embodiment of the invention may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that, for clarity, representations and descriptions of components and processes unrelated to the present invention and known to those skilled in the art have been omitted from the drawings and description.

[0028] This invention provides a method for storing large-scale relational data and performing paginated queries based on HBase. The core components include composite RowKey design, variable weight encoding range query, column families and dynamic columns, consistent pagination mechanism, 7-byte compressed token, sequential index to eliminate offset holes, index-main table merged Bloom filter, online hotspot rehashing, and multi-level caching and prefetching.

[0029] 1. Composite RowKey Design It adopts a six-segment structure: "Node Type A | Node Value A | Node Type B | Node Value B | Relationship Edge Type | Timestamp (milliseconds)" and follows the order of "node type priority" (e.g., USER < PRODUCT < CATEGORY < SHOP). Only one row needs to be stored for the same logical edge, which can be used to complete bidirectional queries through forward or reverse Scan, avoiding redundant rows in traditional solutions.

[0030] 2. Variable weight encoding range query Append 1-2 bytes of Gamma-encoded weight value after the fifth segment "Relationship Edge Type" of the composite RowKey to form "Starting node type | Starting node identifier | Target node type | Target node identifier | Relationship edge type | gamma (weight) | Relationship generation timestamp"; When querying the weight range [1000, 2000], only the byte boundaries of gamma(1000) and gamma(2000) need to be calculated, and StartRow / StopRow can be set directly without Filter; in the scenario of 120 billion edges, the average latency of Scan in the weight range is reduced from 800ms to 50ms, and the storage volume increases by <2%.

[0031] 3. Column families and dynamic columns The info column family stores edge attributes (weight, state, creation time, etc.), and uses VERSIONS=1 to control the retention strategy. Meta column family: Stores system maintenance fields (lastModifyTime, checksum, lockFlag, etc.); It supports dynamic column qualifiers, allowing businesses to add attributes at any time without modifying the table structure. After column names are compressed by Snappy, the storage overhead is less than 1%.

[0032] 4. Consistent pagination mechanism a) Before initiating the first round of queries, the client generates a global snapshot version number T0 (millisecond-level timestamp) and persists it to Redis along with the page size and the starting RowKey as a pagination token; b) Each time an HBase Scan is constructed, the visibility is fixed at time T0 by settingTimeRange(0,T0). Combined with MVCC's readPoint, this ensures that subsequent writes, deletions, and updates are not visible to this page-turning session. c) Use "RowKey of the last item on the previous page + ε" as the StartRow for the next page, and set StopRow to the maximum boundary of the business logic; use PageFilter to limit the number of items returned per page; ε is the minimum offset; d) If the session has not ended after 60000ms, the client calls the / renew interface to take a new snapshot at T0+1ms and return a new token to achieve short transaction renewal.

[0033] 5-7 byte compressed token It uses a 42-bit millisecond timestamp + 22-bit cursor offset to form 56-bit data, which is only 7 bytes after Base64 encoding; when renewing the lease, only delta=1ms is transmitted. Compared with plaintext T0 + lastRowKey, the token size is reduced by 50%, and the renewal RPC QPS is increased by 3 times (in a real test scenario of 1 billion pages / day).

[0034] 6. Sequential indexing eliminates offset holes For each relation edge, asynchronously double-write a single-row sequential index, with the row key format as follows: {2-digit hex shard}{12-digit Snowflake serial number}{Original RowKey} Paging scans are performed only on sequential indexes. Page boundaries are located by consecutive intervals of seqId, and then the original table is retrieved in batches, completely avoiding offset gaps caused by "intermediate insertions". In a scenario with 120 billion edges, the average latency of the 500,000th page is 28ms, P9945ms, and the storage overhead increases by 4.8% (sequential index).

[0035] 7. Index-Main Table Merge Bloom Filter By appending a 3-byte segmented Bloom to the end of the sequential index row key and performing local filtering before batch Get operations, the number of empty Get operations is reduced. In a scenario with 120 billion edges, the empty Get rate is reduced from 12% to 0.3%, and the total latency is reduced by another 15%.

[0036] 8. Online Hotspot Redistribution RegionServer provides real-time QPS statistics. When the QPS of a single Region exceeds the threshold, it triggers an online split with "prefix bit + 1" without stopping writes. The QPS of hot regions decreased from 120,000 to 30,000, and the peak write speed of the cluster increased by 40%.

[0037] 9. Multi-level caching and prefetching Enable BlockCache+BucketCache and increase the cache ratio to 0.5; The business-side local Caffeine cache key format is: userId:pageNo:{T0 / / 60000ms}, which automatically expires after 60000ms to ensure that the cache is aligned with the snapshot view; When writing sequential indexes, 1 bit of cache invalidation is written synchronously, and the cache is cleared in batches when renewing the lease to avoid cross-node invalidation storms; cache invalidation latency is reduced from >200ms to <5ms.

[0038] Comparative Example Using the traditional LIMIT-OFFSET scheme, the average time for 120 billion edges and page 500,000 is 42s, and for P99 it is 61s. After adopting this embodiment, the average time for the same page number is 28ms, and for P99 it is 45ms, with a 4.8% increase in storage overhead (sequential index).

[0039] For details, see Figure 1 The method for large-scale relational data storage and paginated query based on HBase of the present invention includes: Step a) Use a composite row key structure to normalize and store relational data. The row key includes: starting node type (node ​​type A), starting node identifier (node ​​value A), target node type (node ​​type B), target node identifier (node ​​value B), relation edge type, and relation generation timestamp, which can uniquely identify a relation edge and support time-series queries. The composite row key is used as a whitelist key for setTimeRange(0,T0) in subsequent step d), and is directly used as the boundary parameter between StartRow and StopRow in step e to locate the scan interval. Step b) Sort the row keys according to the preset node type priority. The node with higher priority is always placed on the left side of the row key to achieve bidirectional traversal without redundant storage, thereby improving storage efficiency and query performance. The node type priority and sorting result are applied to both the main table write and the sequential index construction in step i, ensuring that the relative positions of the nodes at both ends in the row key are fixed during bidirectional queries, without the need for redundant reverse writing. Step c) Construct a column family structure, which includes an information column family (info column family) for storing basic attributes of relationships, a metadata column family (meta column family) for storing system metadata, and supports dynamic column expansion to adapt to flexible changes in business attributes; The information column family is returned to the client in the scan results of step e; the metadata column family is used to store lastModifyTime during lease renewal in step h) to determine whether a new snapshot needs to be generated. Step d) Before the first query, the client generates a global snapshot version number T0, and fixes the data visibility in all subsequent pagination requests by using the time range (0,T0) (setTimeRange(0,T0)). Combined with the MVCC (Multi-Version Concurrency Control) mechanism, the data consistency (i.e., pagination result consistency) during the pagination query process is guaranteed. The T0 is simultaneously written to the cache key in step j and the new token in step h, so that when the cache is hit, the T0 truncation value is compared first. If the minute part is different, it is forcibly invalidated, ensuring that the cache view is consistent with the snapshot. Step e) Construct paginated queries based on a range scan of the sequential index row keys. When the system enables sequential indexes, the scan target in this step is switched from composite row keys to sequential index row keys, and the main table is retrieved in batches using the original row keys returned by the sequential index. The start and end row keys are set to limit the scan range. The last sequential index row key returned in the previous step is used as the starting position for the next scan, and the main table is retrieved in batches using the original row keys contained within it. Page filtering (e.g., limit=20) limits the number of rows returned per page, ensuring that the deep pagination latency remains stable at the millisecond level. The last row key is taken from the composite row key defined in step a; the limit value of the page filter parameter PageFilter is linked to the local cache capacity strategy in step g, and asynchronous prefetching is triggered when the remaining cache entries are less than the limit; Step f) Use hash prefix or string reversal strategies for high-frequency node identifiers to avoid region hotspots and improve the concurrency performance of writes and queries; The hashed node identifier directly replaces the "starting node identifier" or "target node identifier" field in step a, and is also applied to the sharding prefix of the sequential index in step i, ensuring that the main table and the index table use the same hashing rules. Step g) Utilize block cache, bucket cache and business local cache to build a multi-level caching system, combined with asynchronous prefetching mechanism to accelerate access to hot data and reduce query latency.

[0040] The block cache and bucket cache are used to accelerate the sequential index scan in step e; the key-value pairs of the business local cache are generated in step j, and the cache invalidation policy is consistent with the T0 renewal period in step h.

[0041] Step h) limits the snapshot validity period to no more than 1 minute. When the pagination session expires, the snapshot is regenerated at T0+1ms and a new token is returned. T0 in the new token is written back to the setTimeRange parameter in step d and synchronously updated to the cache key in step j, realizing "short transaction renewal" without rebuilding the scanned data; Step i) Asynchronously writes a row of sequential index to each relation edge. The row key of the sequential index consists of a sharding prefix, a monotonically increasing sequence number, and the original row key. Page scanning is performed only on the sequential index. The sharding prefix shares the same hash function as the hash prefix in step f. The monotonically increasing sequence number is used for continuous positioning of the StartRow in this step's sequential index scan to eliminate offset gaps caused by intermediate insertions. Using the list of original row keys parsed from the sequential index as input, the main table is read in batches to obtain the information column family data from step c. Step j) The cache key includes the user identifier, page number, and a 60000ms (1 minute) truncation value for T0 to ensure consistency between the cache view and the snapshot view. The cache key format can be set to userId:pageNo:{T0 / / 60000}, automatically expiring across minutes to avoid returning expired snapshots. The 60000ms (1 minute) truncation value aligns with the renewal period in step h, forcibly invalidating the cache across minutes to prevent returning expired snapshots.

[0042] It uses a 42-bit millisecond timestamp + 22-bit cursor offset to form 56-bit data, which is only 7 bytes after Base64 encoding; during lease renewal, only delta=1ms is transmitted. Compared with plaintext T0 + lastRowKey, the token size is reduced by 50% and the lease renewal RPC QPS is increased by 3 times. Step k) Append 1-2 bytes of Gamma-encoded weight value after the fifth segment "Relationship Edge Type" of the composite row key to form "Starting Node Type|Starting Node Identifier|Target Node Type|Target Node Identifier|Relationship Edge Type|gamma(weight)|Relationship Generation Timestamp"; When querying the weight range [1000,2000], only the byte boundaries of gamma(1000) and gamma(2000) need to be calculated, and StartRow / StopRow can be set directly without filtering; In the scenario of 120 billion edges, the average latency of Scan in the weight range is reduced from 800ms to 50ms, and the storage volume increases by <2%.

[0043] By appending a 3-byte segmented Bloom filter to the end of the sequential index row key, local filtering is performed before batch Get operations to reduce empty Get operations; in a scenario with 120 billion edges, the empty Get rate is reduced from 12% to 0.3%, and the total latency is reduced by another 15%. RegionServer provides real-time QPS statistics. When the QPS of a single Region exceeds the threshold, it triggers an online split with "prefix bit + 1" without stopping writes. The QPS of hot regions decreased from 120,000 to 30,000, and the peak write volume of the cluster increased by 40%.

[0044] In the short transaction renewal step, data versions earlier in the time range (0, T0-60000ms) are allowed to be physically reclaimed to reduce the risk of version backlog.

[0045] The sequential index uses the Snowflake algorithm to generate sequence numbers, ensuring that the global sequence is monotonically increasing and sortable.

[0046] Node type priority and lexicographical order rules are configured at the table level and can be dynamically adjusted according to business needs. After adjustment, there is no need to rebuild the index for existing data.

[0047] Application Scenario 1: Social Network Friend List Create a table named "relation" with 256 pre-partitions, and use SPLITS files named 00, 01, ... ff; Friend RowKey: USER|{hash3(userIdA)}|USER|{userIdB}|FRIEND|gamma(weight)|{timestamp}; BufferedMutator is used for writing, with 100,000 records per batch and asynchronous flushing; Example of a page-turning token: {T0=1658035200000,pageSize=20,lastRow=…}; Server-side Scan code construction: scan.setTimeRange(0,T0); scan.setStartRow(lastRow+0x00); scan.setStopRow(USER|{userIdA}|~); scan.setFilter(new PageFilter(20)); Return 20 records and a new lastRow, the client updates the token, and the loop continues until there is no more data.

[0048] Weighted range query: For a query of weight range [1000, 2000], calculate the byte boundaries of gamma(1000) and gamma(2000) and directly set StartRow / StopRow; in a scenario with 120 billion edges, the average latency is 50ms and the storage increase is <2%.

[0049] 7-byte compressed token renewal: It uses a 42-bit millisecond timestamp + 22-bit cursor offset to form 56-bit data, which is only 7 bytes after Base64 encoding; when renewing the lease, only delta=1ms is transmitted. Compared with plaintext T0 + lastRowKey, the token size is reduced by 50%, and the renewal RPC QPS is increased by 3 times (in a real test scenario of 1 billion pages / day).

[0050] Sequential indexing eliminates offset holes: For each relation edge, asynchronously double-write a single-row sequential index, with the row key format as follows: {2-bit hex shard}{12-bit Snowflake serial number}{original RowKey}; Paging scans are performed only on sequential indexes. Page boundaries are located by consecutive intervals of seqId, and then the original table is retrieved in batches, completely avoiding offset gaps caused by "intermediate insertions". In a scenario with 120 billion edges, the average latency of the 500,000th page is 28ms, P9945ms, and the storage overhead increases by 4.8% (sequential index).

[0051] Index - Main Table Merge Bloom Filter: By appending a 3-byte segmented Bloom to the end of the sequential index row key and performing local filtering before batch Get operations, the number of empty Get operations is reduced. In a scenario with 120 billion edges, the empty Get rate is reduced from 12% to 0.3%, and the total latency is reduced by another 15%.

[0052] Online hotspot rehashing: RegionServer provides real-time QPS statistics. When the QPS of a single Region exceeds the threshold, it triggers an online split with "prefix bit + 1" without stopping writes. The QPS of hot regions decreased from 120,000 to 30,000, and the peak write speed of the cluster increased by 40%.

[0053] Multi-level caching and prefetching: Enable BlockCache+BucketCache and increase the cache ratio to 0.5; The business-side local Caffeine cache key format is: userId:pageNo:{T0 / / 60000ms}, which automatically expires after 60000ms to ensure that the cache is aligned with the snapshot view; When writing sequential indexes, 1 bit of cache invalidation is written synchronously, and the cache is cleared in batches when renewing the lease to avoid cross-node invalidation storms; cache invalidation latency is reduced from >200ms to <5ms.

[0054] Test results for 200 RegionServers, 120 billion edges, and a page size of 20: Page 10,000: Average 18ms, P99 32ms; Page 100,000: Average 22ms, Page 99 38ms; Page 500,000: Average 28ms, P99 45ms.

[0055] Storage overhead: Sequential indexes consume an additional 4.8% of disk space, and write QPS decreases by less than 3%.

[0056] Weighted range query comparison: Traditional Filter: Average 800ms; The Gamma encoding of this invention has an average duration of 50ms.

[0057] Comparison of cache expiration latency: Traditional cross-node failure: average >200ms; The average Bloom filter bit of this invention is <5ms.

[0058] Application Scenario 2: Knowledge Graph Entity Query The node type is expanded to PERSON, MOVIE, and CITY, with priority PERSON < MOVIE < CITY; when storing the "actors appearing in movies" relationship: RowKey: PERSON|{personId}|MOVIE|{movieId}|ACTED_IN|{ts}; Reverse lookups simply require swapping the positions of the two ends; attribute columns use FST compression, saving 30% of storage.

[0059] Application Scenario 3: Financial Risk Control Transaction Network For transactions with large amounts and numerous attributes, dynamic columns such as info:amount, info:channel, and info:location are used. The SHORT_ZIP compression algorithm is used, achieving an average compression rate of 42% for string columns. Risk control rules require backtracking all transaction paths within the past 30 days. This solution can obtain a 30-day snapshot with a single setTimeRange(0, 30_days_ago), eliminating the need for offline table import. Even with deep pagination up to 500,000 pages, the average latency remains at 28ms.

[0060] Example 4: Snapshot Renewal Code Snippet The client stores the token {T0, lastRowKey, expireAt} in Redis; when expireAt - now < 10s, a background thread asynchronously calls the following code: long newT0=System.currentTimeMillis(); Token newToken=new Token(newT0,lastRowKey,now+60_000); The server returns a new token, which the client atomically replaces, and the original token immediately becomes invalid, ensuring that snapshots are not lost and versions can be recycled in a timely manner.

[0061] Existing technologies relying solely on RowKey-cursor pagination still suffer from data drift during deep pagination. To address this, this application locks visibility at millisecond T0 using `setTimeRange(0,T0)` before the first Scan is opened; it also sends the T0 page token back to the client; subsequent paginations reuse the same T0; and the server directly filters out new writes exceeding T0, even if their sequenceId is less than or equal to the current readPoint. This fixes the "global snapshot" at the moment of turning the first page, preventing subsequent inserts, deletions, and updates from entering the current pagination session and completely eliminating drift. Therefore, this application employs a composite RowKey to achieve bidirectional queries while simultaneously resolving the weighted range Scan problem.

[0062] Meanwhile, this application addresses the aforementioned drift issue by incorporating an MVCC mechanism combined with TimeRange. However, the MVCC mechanism introduces new challenges to pagination: the inherent trade-off between "fixed snapshots" and "dynamic offsets." In engineering practice, this often necessitates the use of multiple compensation mechanisms, such as client-side deduplication, cursor renewal, snapshot renewal, and boundary checks, to balance consistency and performance in deep pagination scenarios involving large datasets. To overcome this, this application minimizes the "snapshot lifecycle" and removes the "pagination state" from the database. In other words, while proposing MVCC snapshot pagination, this application optimizes the renewal token size to resolve the incompatibility between "fixed snapshots" and "dynamic offsets." Furthermore, this application uses sequential indexes to eliminate offset gaps and introduces an index-main table Bloom filter; it employs hash prefixes to distribute hotspots, achieving online hotspot rehashing. The solution can be applied to petabyte-scale scenarios, achieving millisecond-level response times for deep pagination, making it suitable for relational data and knowledge graph analysis scenarios in various industries' big data projects.

[0063] The specific scenarios covered by this application include, but are not limited to: 1. Social networks: friend relationship storage and paginated display, supporting a network of hundreds of millions of user relationships; 2. Knowledge graphs: entity relationship storage and traversal, enabling efficient graph querying; 3. Recommendation systems: user-item relationship storage and real-time recommendation; 4. Financial risk control: transaction relationship network analysis and health monitoring; 5. Internet of Things: device relationship management and efficient querying.

[0064] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0065] Furthermore, the method of the present invention is not limited to being executed in the chronological order described in the specification, but may also be executed in other chronological orders, in parallel, or independently. Therefore, the execution order of the method described in this specification does not constitute a limitation on the technical scope of the present invention.

[0066] Although the invention has been disclosed above through the description of specific embodiments, it should be understood that all the embodiments and examples described above are exemplary and not restrictive. Those skilled in the art can design various modifications, improvements, or equivalents to the invention within the spirit and scope of the appended claims. These modifications, improvements, or equivalents should also be considered to be included within the protection scope of the invention.

Claims

1. A method for large-scale relational data storage and paginated query based on HBase, characterized in that, Includes the following steps: a) A composite row key structure is used to standardize and store relational data. The composite row key includes: starting node type, starting node identifier, target node type, target node identifier, relation edge type, and relation generation timestamp. b) Sort the row keys according to the preset node type priority, with the higher priority row key always placed to the left of the row key to achieve bidirectional traversal without redundant storage; the node type priority and sorting result are applied to both the main table write and subsequent sequential index construction. c) Construct information column families to store basic relationship attributes, and metadata column families to store system metadata and support dynamic column qualifier extensions; the information column families are used to be returned to the client from the scan results; the metadata column families are used to store the last modification time when renewing short transactions to determine whether a snapshot needs to be regenerated.

2. The method for large-scale relational data storage and pagination query based on HBase according to claim 1, characterized in that, It also includes the following steps: d) Before the first query, the client generates a global snapshot version number T0. In subsequent pagination requests, visibility is locked by setting a time range (0, T0), and the consistency of pagination results is ensured by combining the MVCC mechanism. The whitelist key for setting the time range (0, T0) is a composite row key. T0 is written to both the cache key and the new token, so that when the cache is hit, the T0 truncation value is compared first. If the minute part is different, it is forcibly invalidated, ensuring that the cache view is consistent with the snapshot. The unit of T0 is milliseconds. The cache key includes a user identifier, page number, and a minute-level truncation value for T0 to ensure consistency between the cache view and the snapshot view; the minute-level truncation value is aligned with the short transaction renewal period. e) A paginated query is constructed based on a range scan of the sequential index row key. The starting and ending row keys are set to limit the scan range. The last row key returned in the previous scan is used as the starting position of the next scan. The original row key contained in the previous scan is used to perform batch lookups back to the main table. With page filtering to limit the number of pages returned, the latency of deep pagination is stabilized at the millisecond level. The boundary parameters of the starting and ending row keys, as well as the last row key, are all obtained from the sequential index row key. f) Use hash prefix or string reversal strategies for high-frequency node identifiers to avoid regional hotspots; directly replace the starting node identifier or target node identifier field in step a with the node identifier after hash prefix or string reversal, and apply it to the sharding prefix of the subsequent sequential index to ensure that the main table and the index table use the same hashing rules.

3. The method for large-scale relational data storage and pagination query based on HBase according to claim 2, characterized in that, It also includes the following steps: g) Construct a multi-level caching system using block cache, bucket cache, and business local cache, and use asynchronous prefetching to reduce query latency; the block cache and bucket cache are used to accelerate sequential index scanning; the business local cache is linked to the minimum value of the page filtering parameter in step e, and asynchronous prefetching is triggered when the remaining entries in the business local cache are less than the minimum value of the page filtering parameter.

4. The method for large-scale relational data storage and pagination query based on HBase according to claim 3, characterized in that, It also includes the following steps: h) Limit the snapshot validity period to 1 minute. When the page turning session expires, regenerate the snapshot with T0+1ms and return a new token to realize short transaction renewal. T0 in the new token is written back to the set time range parameter in step d and synchronously updated to the cache key to realize short transaction renewal without rebuilding the scanned data. The cache key is forcibly invalidated when the minutes are crossed to prevent the return of expired snapshots.

5. The method for large-scale relational data storage and pagination query based on HBase according to claim 4, characterized in that, It also includes the following steps: i) Write a sequential index row asynchronously for each relation edge. The sequential index row key consists of a sharding prefix, a monotonically increasing sequence number and the original row key. Page scanning is performed only on the sequential index. The sharding prefix and the hash prefix in step f share the same hash function; the monotonically increasing sequence number is used for the continuous positioning of the starting row key of the sequential index scan to eliminate offset gaps caused by intermediate insertions; and the original row key list parsed from the sequential index is used as input to perform batch reading of the main table to obtain the information column family data in step c.

6. The method for large-scale relational data storage and paginated query based on HBase according to claim 4, characterized in that, In step h, data versions prior to the time range (0, T0-60000ms) are allowed to be physically reclaimed to reduce the risk of version backlog.

7. The method for large-scale relational data storage and paginated query based on HBase according to claim 5, characterized in that, The monotonically increasing sequence number in step i is generated by the Snowflake algorithm, and the sequence number is consecutive in lexicographical order within the row key.

8. The method for large-scale relational data storage and pagination query based on HBase according to claim 2, characterized in that, The cache key is in the format userId:pageNo:{T0 / / 60000ms} and expires automatically after a minute.

9. The method for large-scale relational data storage and pagination query based on HBase according to claim 1, characterized in that, The composite row key also includes a 1-2 byte Gamma-encoded weight value, forming "starting node type|starting node identifier|target node type|target node identifier|relation edge type|gamma(weight)|relation generation timestamp"; where gamma(weight) is a variable-length encoding of the weight value.

Images