Database re-sharding method, database platform, and computing device

Abstract

Embodiments of the present specification provide a database reshuffling method, a database platform and a computing device. The method comprises: in response to a database reshuffling request, creating a number of new shards corresponding to the reshuffling request; in response to an in-memory table switching instruction of an old shard, marking an old in-memory table of the old shard as a frozen state, creating a writable in-memory table on the old shard, and starting an asynchronous disk flushing task of the old in-memory table, the asynchronous disk flushing task being used for asynchronously writing data of the old in-memory table to a disk file; routing a second write request and subsequent write requests to the writable in-memory table of the old shard and in-memory tables of new shards; in response to a shard switching event, routing a third write request to the in-memory table of the new shard, and releasing the old shard after read request processing on the old shard is completed. The old in-memory table of the old shard is frozen in a request gap, the old in-memory table is asynchronously flushed, and double writing is started, the old shard and the new shard simultaneously receive write, data is not lost, and data continuously flows.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of database, and particularly relate to a database resharding method, a database platform and a computing device. BACKGROUND

[0002] With the rapid development of computer technology and network technology, database technology also develops rapidly, and has the characteristics of multiple hardware resources, high processing performance, and the ability to meet real-time analysis of big data requirements, and is widely used in many industries and fields. In high-performance databases, data is usually divided into multiple shards to achieve horizontal expansion. Among them, the in-memory table (MemTable) as the core component of receiving real-time writes, constitutes the real-time read-write layer of the database, and when the data accumulates to a certain extent, the contents in the in-memory table will be asynchronously flushed to persistent disk files.

[0003] In the prior art, when the shard structure of the database is adjusted, the write is often suspended, and the write request is switched to the new shard after the in-memory table of the old shard is completely flushed, which causes service interruption or significantly increases the write delay. Therefore, there is an urgent need for a database resharding solution that does not affect the write service. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a database resharding method. One or more embodiments of the present specification also relate to a database platform, a computing device, a computer readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a database resharding method is provided, comprising:

[0006] In response to a database resharding request, a number of new shards corresponding to the resharding request is created;

[0007] In response to an in-memory table switching instruction of an old shard, an old in-memory table of the old shard is marked as a frozen state, a writable in-memory table is created in the old shard, and an asynchronous flushing task of the old in-memory table is started, wherein the in-memory table switching instruction of the old shard is generated after the first write request is processed and before the second write request is processed, and the asynchronous flushing task is used to asynchronously write data of the old in-memory table to a disk file;

[0008] The second write request and subsequent write requests are routed to the writable in-memory table of the old shard and the in-memory table of the new shard;

[0009] In response to a shard switching event, the third write request is routed to the in-memory table of the new shard, and the old shard is released after the read request processing on the old shard is completed.

[0010] According to a second aspect of the embodiments of the present specification, a database platform is provided, comprising:

[0011] A creating module configured to create a number of new shards corresponding to a re-sharding request of a database in response to the re-sharding request;

[0012] A freezing module configured to mark an old in-memory table of an old shard as a frozen state, create a writable in-memory table on the old shard, and start an asynchronous disk flushing task of the old in-memory table in response to an in-memory table switching instruction of the old shard, wherein the in-memory table switching instruction of the old shard is generated after a first write request is processed and before a second write request is started to be processed, and the asynchronous disk flushing task is used to asynchronously write data of the old in-memory table to a disk file;

[0013] A double writing module configured to route the second write request and subsequent write requests to the writable in-memory table of the old shard and the in-memory table of the new shard;

[0014] A switching module configured to route a third write request to the in-memory table of the new shard in response to a shard switching event, and release the old shard after read request processing on the old shard is completed.

[0015] According to a third aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0016] A memory and a processor;

[0017] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the re-sharding method of the database.

[0018] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions, when executed by a processor, implement the steps of the re-sharding method of the database.

[0019] According to a fifth aspect of the embodiments of the present specification, a computer program product is provided, comprising computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the re-sharding method of the database.

[0020] One embodiment of the present specification provides a database re-sharding method, in response to a database re-sharding request, creating a number of new shards corresponding to the re-sharding request; in response to an old shard memory table switching instruction, marking an old memory table of the old shard as a frozen state, creating a writable memory table on the old shard, and starting an asynchronous disk flushing task of the old memory table, wherein the old shard memory table switching instruction is generated after a first write request is processed and before a second write request is started, and the asynchronous disk flushing task is used to asynchronously write data of the old memory table to a disk file; routing the second write request and subsequent write requests to the writable memory table of the old shard and the memory table of the new shard; in response to a shard switching event, routing a third write request to the memory table of the new shard, and releasing the old shard after read request processing on the old shard is completed.

[0021] One embodiment of the present specification realizes that when it is detected that the number of shards of a database needs to be adjusted, a new shard is dynamically created, in response to an old shard memory table switching instruction, an old memory table of the old shard is frozen, a writable memory table is created on the old shard, and an asynchronous disk flushing task of the old memory table is started, data in the old shard that has not been written to disk is gradually written to a persistent disk file, for each write request initiated after the memory table switching instruction (i.e., the second write request and subsequent write requests), the old shard writable memory table and the new shard memory table are written simultaneously, double writing is realized, in response to a shard switching event, all subsequent write requests (i.e., the third write request) are routed to the memory table of the new shard, and the old shard is released after read request processing on the old shard is completed, and dynamic re-sharding of the database is completed. In this way, by actively freezing the old memory table of the old shard in the request gap, asynchronously flushing the old memory table, and starting double writing, the old and new shards simultaneously receive write requests, ensuring that data is not lost and continuously flows, ensuring that the switching process is transparent to the upper layer, realizing uninterrupted write requests during the shard switching process, realizing write-while-expand, avoiding the problem of long-time waiting for the old shard to flush or direct stop writing, that is, providing the ability to dynamically adjust the database shards in the continuous writing process, the entire process does not need to be stopped and does not interrupt the service, supporting the elastic scaling of database shards in a high-concurrency scenario, ensuring high availability, improving the flexibility and reliability of the system, and significantly improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a database re-sharding method provided by one embodiment of the present specification;

[0023] Figure 2 is a processing process diagram of a database re-sharding provided by one embodiment of the present specification;

[0024] Figure 3This is a flowchart illustrating the processing procedure of a database resharding method provided in one embodiment of this specification;

[0025] Figure 4 This is a schematic diagram of the structure of a database platform provided in one embodiment of this specification;

[0026] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0027] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0030] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0032] MemTable: A memory structure used to receive real-time written data. It is equivalent to a high-speed cache. All newly added or modified data is first written here and then asynchronously saved to the disk file. It is the core component for achieving high-throughput writing.

[0033] Frontend: The request access and scheduling module is responsible for receiving user read and write requests, routing them to the corresponding shards, and coordinating dual writes and traffic switching during the resharding process to ensure data consistency and service continuity.

[0034] Sharding: Data is split into multiple independently managed units according to certain rules. Each unit is called a shard. Shards can be distributed across different nodes to improve concurrent processing capabilities and storage capacity.

[0035] Resharding: When the amount of data or the load changes, the number or distribution of shards is dynamically adjusted, such as expanding 4 shards to 8, or merging them into 2, with the aim of maintaining a balanced database performance.

[0036] Dual Writing: During the resharding process, the front end simultaneously sends newly written data to both the old and new shards to ensure that no data is lost during the switchover. This is a key mechanism for achieving uninterrupted migration.

[0037] Logical location identifier: An abstract number used to uniquely identify a data record, such as a globally incrementing sequence number. Deletion operations are based on this identifier, regardless of which file or line the data is stored in.

[0038] Physical location: The actual storage address of the data on the disk, such as which file, line group, and line number it belongs to. The deletion mechanism relies on this information, but it is easily invalidated by file changes.

[0039] Deletion Vector: A structure used to mark whether a data row has been deleted. It is commonly used in columnar storage systems and is used in conjunction with logical position identifiers to quickly determine whether a record has been deleted during a query.

[0040] Flush: The process of writing data from the MemTable that has not yet been written to disk to a disk file is an important step in data persistence.

[0041] Immutable MemTable: Once frozen, a MemTable no longer accepts writes and only waits for the disk flush to complete. It is used to ensure clear data boundaries and consistent state during the switchover process.

[0042] It should be noted that in high-performance databases, data is typically divided into multiple shards to achieve horizontal scaling. Among them, the MemTable, as the core component for receiving real-time writes, constitutes the real-time read / write layer of the system. When the data accumulates to a certain level, the contents of the MemTable are asynchronously flushed to disk as persistent files.

[0043] In one alternative implementation, adjusting the sharding structure often requires pausing writes and waiting for the MemTable in the old shards to be completely flushed to disk before switching traffic. While this method ensures consistency, it leads to write service interruptions, making it impossible to support 24 / 7 real-time writing, which can result in service outages or significantly increased write latency. Another alternative implementation can achieve online migration through asynchronous replication or log synchronization, but this typically only applies to files already written to disk, neglecting the state synchronization issue of the in-memory table—a real-time read / write layer—leading to data loss or invalidation of deletion operations during switching. Therefore, how to achieve dynamic sharding adjustment without affecting real-time writing and ensure consistency of operations across various storage modes is a key challenge in current technology.

[0044] Therefore, this specification provides a database resharding scheme. By freezing the old MemTable and initiating dual writes during user request intervals, it ensures that the switching process is seamless for the upper layer. Furthermore, logical location identifiers can be used instead of physical location record deletion operations, ensuring that deletion semantics remain accurate after shard migration and avoiding data residue. It provides the ability to dynamically adjust data shards during continuous real-time writing, solving the problem of needing to stop writing before scaling. The entire process does not require interrupting writing, supporting elastic scaling in high-concurrency real-time scenarios and significantly improving the user experience.

[0045] This specification provides a database resharding method, and also relates to a database platform, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments.

[0046] See Figure 1 , Figure 1 A flowchart of a database resharding method according to an embodiment of this specification is shown, specifically including steps 102-108.

[0047] Step 102: In response to the database's resharding request, create the number of new shards corresponding to the resharding request.

[0048] It should be noted that the database resharding scheme provided in the embodiments of this specification can be applied to a variety of storage architectures, including read-time merging architecture based on LSM Tree (Log-Structured Merge Tree) and write-time merging architecture based on Deletion Vector.

[0049] LSM Tree is a storage structure designed for high write throughput and is widely used in database and storage systems. It adopts a "write log first, then merge" strategy. Through "append write", all modification operations (insert, update, delete) are first written to the sharded memory table, and then batch sequentially flushed to disk as disk files (SSTable files). Since the same key may exist in multiple versions in multiple levels of disk files, the actual "merging" is not done during writing, but dynamically during querying, that is, "merge on read". It sacrifices some read performance for extremely high write efficiency and is suitable for scenarios with many writes and few reads.

[0050] A Deletion Vector is a metadata structure used for efficient management of batch deletions, commonly found in columnar databases. It records which rows (row IDs) have been logically deleted. The data is stored in columnar format, and the deletion operation does not modify the original file; instead, the row numbers to be deleted are recorded in a Deletion Vector (essentially a bitmap or a list of integers).

[0051] Specifically, sharding refers to dividing data into multiple independently managed units according to certain rules. Each unit is called a shard, and shards can be distributed across different nodes; in other words, a shard is an independently managed data unit. Resharding is the dynamic adjustment of the number or distribution of shards in the real-time read / write layer of a database when data volume or load changes. A resharding request is an instruction issued by the user or the database system after automatically detecting the need for expansion / shrinkage, requiring adjustments to the database's sharding structure. Creating a new shard refers to initializing new data storage units (including metadata, storage paths, state machines, etc.) on the target node to prepare for subsequent data migration; the number of new shards created corresponds to the number of resharding requests.

[0052] It should be noted that the resharding process remains the same regardless of whether the database storage architecture uses a read-on-demand or write-on-demand architecture. In practice, database resharding requests can be manually triggered by the user, in which case the number of new shards is determined by the user; alternatively, the database can be configured to automatically trigger resharding based on set rules, in which case the number of new shards is determined by the set rules that trigger the resharding request.

[0053] In actual implementation, when the need to adjust the number of shards is detected, the resharding request is correctly parsed and a new set of shards is dynamically created. Each shard corresponds to an independent data management unit (Tablet), and each data management unit contains a memory table (MemTable) structure to support real-time read and write.

[0054] Step 104: In response to the memory table switching instruction of the old shard, mark the old memory table of the old shard as frozen, create a writable memory table in the old shard, and start the asynchronous disk flushing task of the old memory table. The memory table switching instruction of the old shard is generated after the first write request is processed and before the second write request is processed. The asynchronous disk flushing task is used to asynchronously write the data of the old memory table to the disk file.

[0055] Specifically, the old shard refers to the data shard currently receiving write requests, whose memory table is about to be replaced. The old memory table of the old shard refers to the ordered data structure in the old shard that temporarily stores write data. The old shard memory table switching instruction is a control signal that stops writing data to the old memory table of the current old shard. The old shard memory table switching instruction is generated after the first write request is processed and before the second write request is processed. The first write request is the current user's write request (which can be denoted as request A), and the second write request is the next user's write request (which can be denoted as request B). That is, the memory table switching instruction can be initiated by the front-end module to the old shard after completing the processing of the current user's write request and before the processing of the next user's write request, so that the old shard can mark the old memory table of the old shard as frozen, that is, actively freeze the old memory table of the old shard between user requests.

[0056] In this context, "frozen state" refers to marking the old memory table of the old shard as immutable. At this point, the old memory table is no longer accepting new writes and awaits asynchronous flushing to disk. The database then creates a writable memory table for the old shard to receive subsequent dual-write requests. Specifically, the memory table is an ordered memory data structure stored by primary key, and can be based on a skip list, a concurrent hash table + log structure, or dynamically created based on the shard scheduler. The asynchronous flushing task is a background task that persists the data in the frozen old memory table (i.e., the immutable memory table) to a disk file (such as SSTable). This process does not block the main write thread.

[0057] In practice, after the new shard is initialized, the front-end module can send a memory table switch command to the old shard after processing the current user write request (first write request) and before starting to process the next user write request (second write request). In response to the memory table switch command, the old shard marks its currently active old memory table as immutable, that is, freezes the old shard's old memory table, and starts an asynchronous disk flushing task to gradually write the data in the old shard's old memory table that has not yet been written to disk into the persistent disk file.

[0058] It should be noted that the old memory table of the old shard is actively frozen during the interval between user requests to ensure data consistency during the dynamic resharding process, and an asynchronous disk flushing task of the old memory table is started to avoid blocking the write thread and achieve uninterrupted real-time writing during the sharding switching process.

[0059] Step 106: Route the second write request and subsequent write requests to the writable memory table of the old shard and the memory table of the new shard.

[0060] Specifically, the second write request and subsequent write requests refer to all user write requests after the memory table switch instruction is initiated (i.e., all requests starting from request B). The writable memory table of the old slice refers to the memory table in the old slice used to receive write requests during the dual-write phase.

[0061] It should be noted that after the old memory table of the old shard is frozen, it is in a read-only state. At this time, a writable memory table can be created for the old shard to receive write requests in the subsequent dual-write phase. Dual-write means that write requests are written to the memory tables of both the old and new shards at the same time. That is to say, after the memory table switching command is initiated, all user write requests need to be written by the front-end module to both the writable memory table of the old shard and the memory table of the new shard. The old and new shards receive write requests at the same time to ensure that data is not lost and the flow is uninterrupted, avoiding the problem of migration failure due to high traffic.

[0062] In an optional implementation of this embodiment, routing the second write request and subsequent write requests to the writable memory table of the old shard and the memory table of the new shard includes:

[0063] Receive a target write request, wherein the target write request is any one of the second write request and subsequent write requests;

[0064] Write the target write request to the writable memory table of the old shard and write the target write request to the memory table of the new shard, and prevent the old shard and the new shard from starting a new flush task.

[0065] In actual implementation, the target write request is any of the second write request and subsequent write requests. That is, for any write request initiated after the memory table switching instruction, the target write request needs to be written to the writable memory table of the old shard and the target write request needs to be written to the memory table of the new shard. During the dual write phase, both the old shard and the new shard are prohibited from starting new disk flushing operations.

[0066] Specifically, dual-write logic can be configured on the front end. For any write request after the memory table switching instruction is initiated, the front end will route it to both the old shard and the new shard. At the same time, the front end can send instructions to both shards to prevent the start of new disk flushing tasks (such as setting a flag). After the two shards are successfully written, the front end will return a dual-write success message.

[0067] It should be noted that during the dual-write phase, user read requests are still routed to the old shard, which provides the read service.

[0068] In the embodiments described in this specification, after entering the dual-write phase of shard migration, all new write requests are simultaneously written to the memory tables of both the old and new shards to ensure data consistency between the old and new shards. At the same time, both shards are prohibited from starting new disk flushing tasks to prevent data from being persisted before it is fully synchronized, thus avoiding state splitting.

[0069] Step 108: In response to the sharding switch event, route the third write request to the memory table of the new shard, and release the old shard after the read request on the old shard has been processed.

[0070] Specifically, the shard switching event refers to the traffic switching event triggered when the asynchronous disk flushing task of the old shard is completed; the third write request refers to the write request after the shard switching event is triggered. That is, after the asynchronous disk flushing task of the old shard is completed, dual writing stops, and all subsequent user write requests are routed to the new shard. At this time, the old shard enters a read-only state, and the old shard is released after the read requests on the old shard are processed.

[0071] It's important to note that in the final stage of database shard migration or resharding, when switching from dual-write mode to a new shard-independent service mode, all subsequent write requests triggered by the shard switch event are sent to the new shard and written to its in-memory table, no longer written to the old shard. This signifies that the asynchronous disk flushing task of the old shard is complete, and the new shard is now capable of handling write requests. At this point, the old shard enters a read-only state, and is released after all read requests have been processed. Releasing the old shard refers to removing it from the write path, which may include: stopping write reception, closing connections, releasing memory, deleting data files, and reclaiming resources.

[0072] Additionally, the old memory tables (i.e., immutable memory tables) marked as frozen in the old shards are used to initiate asynchronous disk flushing tasks. These tasks are executed asynchronously with the main thread's write requests, persisting data that has not yet been written to disk to disk files. After the asynchronous disk flushing task is completed, the immutable memory tables in the old shards can be destroyed and released. For write requests from the main thread, all write requests after the shard switch event are routed to the new shard, and the old shard enters a read-only state. After the read requests on the old shard are processed, the old shard is destroyed and released, completing the dynamic resharding process of the database.

[0073] In an optional implementation of this embodiment, in response to a shard switching event, routing the third write request to the memory table of the new shard includes:

[0074] Monitor asynchronous disk flushing tasks in older shards;

[0075] When the asynchronous disk flushing task in the old shard is completed, a shard switching event is triggered, and the third write request is routed to the memory table of the new shard, controlling the old shard to enter a read-only state. The third write request is the write request after the shard switching event is triggered.

[0076] Specifically, disk flushing completion means that all data in the old memory table (immutable memory table) of the old shard has been successfully written to the disk file, that is, there is no data in the old shard waiting to be flushed, and the memory can be safely released.

[0077] In actual implementation, it continuously checks whether there are any unfinished asynchronous disk flushing tasks in the old shard to ensure that all data in the old memory table of the old shard has been written to disk. If it is confirmed that the asynchronous disk flushing tasks in the old shard have been completed, it means that there is no data to be flushed to disk in the old shard. At this time, a shard switching event will be triggered, the front end will stop dual writing, and traffic will be switched. All subsequent write requests will be routed to the new shard by the front end, and the old shard will enter a read-only state. After all user read requests on the old shard have been processed, the old shard will be destroyed and released.

[0078] It should be noted that during the dual-write phase, the old shard performs an asynchronous disk flushing task, persisting its data to disk files. All ongoing asynchronous disk flushing tasks in the old shard can be continuously monitored until all are successfully committed. After confirming that the old shard has no data to be flushed (i.e., the asynchronous disk flushing task is complete), a lock is briefly acquired to prevent new transactions from committing. Uncommitted transactions are cancelled and recommitted to the new shard by the upper-layer application.

[0079] Example, Figure 2 This is a schematic diagram illustrating a database resharding process according to one embodiment of this specification, as shown below. Figure 2As shown, the frontend writes write requests to the memory tables of shard 1 and shard 2. Once the data in the memory tables meets the conditions, it persists the data to disk. In response to the database's resharding request, it creates new shards 1, 2, and 3, and marks the old memory tables of shards 1 and 2 as frozen (immutable). It then initiates an asynchronous disk flushing task for these immutable memory tables, persisting them to disk. The frontend then enters a dual-write phase, routing all user write requests after the memory table switching command to the writable memory tables of shards 1 and 2, as well as the memory tables of shards 1, 2, and 3. In response to a sharding switch event, write requests triggered by the sharding switch event are routed to the memory tables of new shard 1, new shard 2, and new shard 3. Old shard 1 and old shard 2 enter a read-only state. Subsequently, if the memory tables of new shard 1, new shard 2, and new shard 3 meet the conditions, persistent file storage is performed, thus completing the dynamic resharding of the database.

[0080] In the embodiments described in this specification, by monitoring the completion status of the asynchronous disk flushing task of the old shard, the memory data in the old shard is ensured to be securely persisted before a shard switching event is triggered. Subsequent write requests are then accurately routed to the new shard, and the old shard is set to read-only to continue serving read requests. The old shard is destroyed and released only after all read operations on it are completed. This ensures data consistency and service continuity while achieving safe and orderly resource reclamation during shard migration, effectively avoiding the risks of data loss, write interruption, and resource leakage.

[0081] It should be noted that although the main process of resharding is the same, when the write request is a delete request, there are fundamental differences in the implementation details of the delete request between the read-time merging architecture and the write-time merging architecture, and adaptation solutions need to be provided for different storage structures.

[0082] In one optional implementation of this embodiment, the database storage architecture is a read-time merging architecture or a write-time merging architecture; when the database storage architecture is a write-time merging architecture, each write record is bound to a corresponding logical location identifier, and the disk file includes a logical location hidden column, which is used to mark the logical location identifier of each write record; the method further includes:

[0083] In response to a deletion request, a deletion record is written into the deletion vector, wherein the deletion record includes a logical location identifier of the data to be deleted;

[0084] Receive a data query request for the target data, wherein the data query request carries a logical location identifier to be queried; query the corresponding target physical location based on the logical location identifier to be queried, and the data query request is triggered based on a query task and / or a disk flushing task.

[0085] In practice, database storage architecture can be divided into read-time merging architecture and write-time merging architecture.

[0086] It should be noted that for a deletion request, the physical location of the data to be deleted on the disk file needs to be known in order to perform the deletion operation. In the read-on-merge architecture, whether it is an insert, update or delete request, it is regarded as a change to the data. The changes all follow the same write path: first write to the memory table, and then flush to the disk file.

[0087] For read-on-merge architecture, data is not physically deleted directly from the disk file. That is, when a user executes a DELETE key operation, the data corresponding to the key is not immediately found and deleted. Instead, a special record called a tombstone is created. The tombstone is usually a record containing the key and a special flag value (such as null or deleted). It is treated as a normal update operation and written to the memory table. When the memory table reaches a certain size, it will be flushed to the disk file. At this time, the tombstone corresponding to the deletion request is also persisted to the disk file along with other data. In other words, for a read-on-merge architecture, delete requests are written to the memory table as tombstone markers and then flushed to the disk file (SSTable file). Since a delete request only writes a tombstone marker, it does not care about the physical location of the original data. During a query, multi-version merging is used to determine whether the data has been deleted. Therefore, in a read-on-merge architecture, during the dual-write phase, delete requests, as part of the data change, will naturally be synchronized to the old and new shards along with the dual-write process. There is no need to handle physical location mapping or cross-shard coordination. Therefore, for a read-on-merge storage architecture, no special mechanism is needed to ensure the correctness of the deletion semantics of the delete request.

[0088] In a read-on-merge architecture, data storage is hierarchical, with each tier potentially containing multiple disk files. When querying a target data, the process begins with the latest in-memory table and proceeds sequentially through the disk files at each level until the latest version of the target data is found. The system then determines whether the corresponding data has been deleted. During the search, if a tombstone marker is found for a target data entry, it confirms that the target data has been deleted.

[0089] In addition, for the write-on-merge architecture, delete requests mark deleted rows by maintaining a delete vector. It is necessary to record the physical location of the data to be deleted in the file (such as file number and line number). The old memory table being flushed to disk in the old shard can also be regarded as a kind of file in memory. If the data to be deleted is data in the old memory table being flushed to disk in the old shard, the new shard cannot access the old memory table being flushed to disk in the old shard, which makes it impossible to complete the physical location lookup, and the consistency of deletion is difficult to guarantee.

[0090] Therefore, logical location identifiers can be introduced for write-on-demand merging architectures. In actual implementation, a corresponding logical location identifier can be bound to each write record. The disk file includes a logical location hidden column, which is used to mark the logical location identifier of each write record. Specifically, a logical location identifier is a globally unique and ordered identifier assigned to each write record, used to uniquely identify the global logical number of a write record. For example, the logical location identifier can be a globally incrementing sequence number, a primary key unique code, etc. The logical location hidden column refers to a field (i.e., a "hidden column") added to each write record in the disk file. This hidden column is specifically used to store the logical location identifier of the write record. In the disk file, the logical location identifier of each write record is associated with its physical location (which file, which line), providing a basis for subsequent fast search and deletion operations.

[0091] In practical implementation, when a write request is a delete request, a delete record can be written to the delete vector. This delete record includes the logical location identifier of the data to be deleted. It's not necessary to know which disk file or line number the data to be deleted is currently stored in; only the logical location identifier is needed. Deleting a record in the delete vector with this logical location identifier indicates that the data corresponding to that logical location identifier has been deleted. Subsequently, when querying whether a target data has been deleted—that is, responding to a data query request for the target data—the target data's current physical location (such as physical file and line number) can be retrieved based on the carried logical location identifier. This allows for finding the corresponding target data and performing appropriate processing. This data query request can be triggered by a query task and / or a disk flushing task to query a target data and determine whether it has been deleted.

[0092] In the embodiments of this specification, logical location identifiers are used to record deletion requests, no longer relying on physical location. This makes deletion requests compatible with shard changes, fundamentally avoiding the complexity of cross-shard reverse lookup of the memory table. It supports online resharding under high concurrency, and ensures the consistency and accuracy of data deletion, achieving write-and-expand, deletion without disorder, and security and reliability.

[0093] In one optional implementation of this embodiment, querying the corresponding target physical location based on the logical location identifier to be queried includes:

[0094] Determine the logical location identifier range for each disk file;

[0095] Based on the logical location identifier range of each disk file, candidate disk files containing the logical location identifier to be queried are determined;

[0096] The logical location identifier to be queried is queried in the candidate disk file to determine the corresponding target physical location.

[0097] Specifically, the logical location identifier range refers to the logical location identifier interval covered by each disk file. For example, if a disk file stores a write record with a logical location identifier of [100, 200), then its logical location identifier range is [100, 200).

[0098] In practice, each disk file maintains metadata indicating the range of logical location identifiers it contains, such as min_log_id and max_log_id. We can first iterate through all possible disk files to determine the range of logical location identifiers for each file. Then, we check if the logical location identifier to be queried falls within the range of a candidate disk file. If it does, it means the data to be queried may exist in the candidate disk file. A precise search is then performed within the candidate disk file to determine the target physical location of the data to be queried.

[0099] For example, suppose there is a logging system that records user operation logs. Suppose a user performs the following insert operations: `INSERT INTO user_logs VALUES (1001, 'user_001', 'login', '2023-01-01 10:00'); INSERT INTO user_logs VALUES (1002, 'user_002', 'login', '2023-01-01 10:01'); INSERT INTO user_logs VALUES (1003, 'user_001', 'upload', '2023-01-01 10:02')`. These records are written to the in-memory table `MemTable`, and each written record has a `log_id` (logical position identifier). If `MemTable` is full, it is flushed to disk to generate the `SSTable-1` file, as shown in Table 1 below.

[0100] Table 1 SSTable-1 file

[0101]

[0102] The metadata of the SSTable-1 file is: min_log_id = 1001, max_log_id = 1003.

[0103] Suppose a user performs a deletion operation, i.e., receives a deletion request: `DELETE FROM user_logs WHERE log_id = 1002`. The system does not directly modify the SSTable-1 file; it only writes the deleted record to the deletion vector: `Deleted Log IDs: {1002}`. Suppose a query request is received: `SELECT * FROM user_logs WHERE log_id = 1002`. Checking the MemTable, there is no record with `log_id = 1002`. The system continues to check the SSTable-1 file, examining its metadata: `min_log_id = 1001`, `max_log_id = 1003`. This confirms that the logical location identifier 1002 is within the logical location identifier range of the SSTable-1 file. The system reads the SSTable-1 file, finds the write record with `log_id = 1002`, obtains the corresponding target physical location, checks the deletion vector, and finds that 1002 is present. The system then returns the deleted query result.

[0104] In the embodiments of this specification, each disk file maintains a range of logical location identifiers contained in the corresponding disk file. Based on the range of logical location identifiers of each disk file, candidate disk files that may contain the data to be queried are filtered, which speeds up the query and realizes efficient indexing and fast retrieval of data query, significantly reducing unnecessary file traversal overhead.

[0105] In one optional implementation of this embodiment, querying the logical location identifier to be queried in the candidate disk file to determine the corresponding target physical location includes:

[0106] An approximate member query structure is used to determine whether the logical location identifier to be queried may exist in the candidate disk file;

[0107] If it is determined that it may exist, then read the logical location identifier of each write record of the candidate disk file and find the target physical location corresponding to the logical location identifier to be queried.

[0108] Specifically, the approximate member query structure is a space-efficient and fast data structure used to determine whether an element "may exist in the set" or "definitely does not exist". Common examples include Bloom filters and Cuckoo filters. It allows a small number of false positives, but does not allow false negatives.

[0109] In one optional implementation, after obtaining candidate disk files through preliminary filtering based on the logical location identifier range, the candidate disk files can be directly read to achieve a reverse lookup of the physical location. In another optional implementation, taking a Bloom filter as an example with an approximate member query structure, each disk file can be associated with a Bloom filter that records the set of all logical location identifiers. After obtaining candidate disk files through preliminary filtering based on the logical location identifier range, the Bloom filter of the candidate disk file can be used to determine whether the logical location identifier to be queried might exist in the candidate disk file. If the Bloom filter returns "definitely not present," the candidate disk file is skipped, and the next disk file is queried. If the Bloom filter returns "possibly present," the logical location identifiers of each write record in the candidate disk file are read to find the target physical location corresponding to the logical location identifier to be queried.

[0110] In the embodiments of this specification, by using an approximate member query structure to quickly filter candidate disk files, it is possible to efficiently determine whether the logical location identifier to be queried may exist, avoiding unnecessary reading of files that do not contain the logical location identifier to be queried, significantly reducing I / O overhead. Under the premise of confirming that the logical location identifier to be queried may exist, the corresponding candidate disk file is loaded and the accurate physical location reverse lookup is achieved, thereby greatly improving query efficiency while ensuring query accuracy.

[0111] In one optional implementation of this embodiment, in response to a deletion request, a deletion record is written into the deletion vector, including:

[0112] When the physical location caching function is enabled, a deletion request is received, the reference physical location of the target data is queried, and a deletion record is written into the deletion vector. The deletion record includes the reference physical location and logical location identifier of the target data.

[0113] The data query request for the target data carries the physical location and logical location identifier of the target data to be queried; after receiving the data query request for the target data, it also includes:

[0114] If the query based on the physical location to be queried fails, then the query based on the logical location identifier to be queried will be executed to find the corresponding target physical location.

[0115] Specifically, the physical location caching function accelerates queries by caching reference physical locations.

[0116] It should be noted that when the physical location caching function is enabled, when receiving a deletion request and writing a deletion record in the deletion vector, it still attempts to look up the reference physical location (file and line number) of the target data (data to be deleted) and record it. The recorded reference physical location is used for caching acceleration.

[0117] In practice, the deleted record can include the reference physical location and logical location identifier of the target data. When the target data needs to be queried later, the corresponding target data can be queried based on the reference physical location of the record. If the data in the reference physical location is not found during the query, the reference physical location is considered invalid, and the query will fall back to the corresponding target physical location using the logical location identifier. The corresponding target data can then be queried based on the target physical location found in the query and processed accordingly.

[0118] In the embodiments of this specification, by recording the reference physical location and logical location identifier of the target data to be deleted when the physical location caching function is enabled, accelerated query based on the reference physical location is realized. During the query process, when the fast access based on the reference physical location fails, it automatically falls back to the search path based on the logical location identifier. This not only utilizes caching to improve performance, but also ensures data consistency through querying based on the logical location identifier, effectively balancing query efficiency and system reliability.

[0119] In an optional implementation of this embodiment, the method further includes:

[0120] Display the runtime configuration page, which includes startup controls for the physical location caching function;

[0121] In response to the trigger operation of the physical location caching function's startup control, the physical location caching function is started.

[0122] Specifically, the runtime configuration page is a graphical or web-based configuration management interface provided at runtime, used to view and modify database parameters such as cache size, feature switches, and log levels.

[0123] In actual implementation, a row configuration page can be displayed, which includes a start control for the physical location caching function. Users can choose whether to enable the physical location caching function. If the user triggers the start control, it means that the physical location caching function is enabled. When writing or deleting a record, the system will still attempt to look up the reference physical location of the target data, thus achieving accelerated querying based on cached physical location. If the user does not trigger the start control, it means that the user has not enabled the physical location caching function. When writing or deleting a record, the system will record the logical location identifier of the target data to be queried, and perform a reverse physical location lookup based on this logical location identifier.

[0124] In this embodiment of the specification, the physical location caching function is visualized and dynamically started / stopped by providing a startup control on the running configuration page. This allows users to choose whether to enable the physical location caching function in order to balance write performance and query efficiency.

[0125] In an optional implementation of this embodiment, the method further includes:

[0126] In response to a disk file merge event, determine the source disk files corresponding to the disk file merge event, wherein the disk file merge event includes a source disk file selection event and / or a commit event after the source disk files have been merged;

[0127] Search the memory table for deletion requests associated with each source disk file;

[0128] If the number of deletion requests exceeds the set threshold, the disk file merge event will be canceled.

[0129] Specifically, a disk file merging event (Compaction) refers to the process of merging multiple disk files into one or more disk files, with the aim of reducing the number of files, improving read performance, and cleaning up expired or deleted data.

[0130] In actual implementation, disk file merge events include source disk file selection events and / or commit events after the source disk file merge is completed. That is, when selecting the source disk file to be merged and when the source disk file is ready to be merged, the deletion requests for the source disk file can be queried in the memory table. If the number of deletion requests exceeds the set threshold, it means that there are too many deletion requests, the disk file merge event is canceled, and the merge is abandoned.

[0131] It should be noted that merging disk files will cause changes in the physical location of the data, which will invalidate the reference physical location cached in memory. Therefore, when selecting the source disk file to be merged and when preparing to commit the merge, the validity of the source disk file can be checked. If it is found that most of the data in the source disk file has been deleted, the merge should be abandoned to avoid invalidating the reference physical location cached in memory, thereby ensuring the query efficiency based on physical location cache.

[0132] One embodiment of this specification implements a method that actively freezes the old memory table of the old shard during request intervals, asynchronously flushes the old memory table to disk, and initiates dual writes. Both the old and new shards simultaneously receive write requests, ensuring no data loss and uninterrupted flow. This ensures the switching process is seamless for the upper layer, achieving uninterrupted write requests during shard switching and enabling simultaneous write and expansion. This avoids the problems of waiting for long periods to flush the old shard to disk or directly stopping writes, providing the ability to dynamically adjust database shards during continuous writing. The entire process requires no downtime and does not interrupt service, supporting elastic scaling of database shards in high-concurrency scenarios. This ensures high availability, improves system elasticity and reliability, and significantly enhances the user experience. Furthermore, it uses logical location identifiers to record deletion requests, eliminating reliance on physical location. This makes deletion requests compatible with shard changes, fundamentally avoiding the complexity of cross-shard reverse lookups of the memory table. It supports online resharding under high concurrency while ensuring the consistency and accuracy of data deletion, achieving simultaneous write and expansion, deletion without data corruption, and security and reliability.

[0133] Figure 3 This document illustrates a flowchart of a database resharding method according to an embodiment of this specification, using a write-on-merge architecture as an example. The process includes the following steps.

[0134] Step 302: In response to the database's resharding request, create the number of new shards corresponding to the resharding request.

[0135] Step 304: After the current user write request is processed and before the next user write request is processed, generate a memory table switching instruction for the old shard, mark the old memory table of the old shard as an immutable memory table, create a writable memory table in the old shard, and start an asynchronous disk flushing task for the immutable memory table of the old shard to write the data of the immutable memory table to the disk file.

[0136] Step 306: Write the target write request (the next user write request that has not yet been processed and all subsequent write requests) to the writable memory table of the old shard, and write the target write request to the memory table of the new shard to achieve dual writing, and prevent the old shard and the new shard from starting a new disk flushing task.

[0137] Step 308: In the dual-write phase after starting the asynchronous disk flushing task of the immutable memory table, monitor whether the asynchronous disk flushing task in the old shard is completed. If the disk flushing is completed, trigger the shard switching event, route the subsequent write requests to the memory table of the new shard, and control the old shard to enter the read-only state; after the read requests on the old shard are processed, destroy and release the old shard.

[0138] Step 310: If the write request is a delete request, in response to the delete request, write a delete record into the delete vector, which includes the logical location identifier of the data to be deleted.

[0139] Step 312: Receive a data query request for the target data, which carries the logical location identifier to be queried; determine the range of logical location identifiers for each disk file; based on the range of logical location identifiers for each disk file, determine candidate disk files containing the logical location identifier to be queried; based on the Bloom filter of the candidate disk files, determine whether the logical location identifier to be queried may exist in the candidate disk file; if it returns that it may exist, read the logical location identifiers of each write record of the candidate disk file and find the target physical location corresponding to the logical location identifier to be queried.

[0140] One embodiment of this specification implements a method that actively freezes the old memory table of the old shard during request intervals, asynchronously flushes the old memory table to disk, and initiates dual writes. Both the old and new shards simultaneously receive write requests, ensuring no data loss and uninterrupted flow. This ensures the switching process is seamless for the upper layer, achieving uninterrupted write requests during shard switching and enabling simultaneous write and expansion. This avoids the problems of waiting for long periods to flush the old shard to disk or directly stopping writes, providing the ability to dynamically adjust database shards during continuous writing. The entire process requires no downtime and does not interrupt service, supporting elastic scaling of database shards in high-concurrency scenarios. This ensures high availability, improves system elasticity and reliability, and significantly enhances the user experience. Furthermore, it uses logical location identifiers to record deletion requests, eliminating reliance on physical location. This makes deletion requests compatible with shard changes, fundamentally avoiding the complexity of cross-shard reverse lookups of the memory table. It supports online resharding under high concurrency while ensuring the consistency and accuracy of data deletion, achieving simultaneous write and expansion, deletion without data corruption, and security and reliability.

[0141] Corresponding to the above method embodiments, this specification also provides database platform embodiments. Figure 4 A schematic diagram of the structure of a database platform provided in one embodiment of this specification is shown. Figure 4 As shown, the database platform includes:

[0142] Create module 402, which is configured to create the number of new shards corresponding to the resharding request in response to a database resharding request;

[0143] The freeze module 404 is configured to, in response to the memory table switching instruction of the old shard, mark the old memory table of the old shard as frozen, create a writable memory table in the old shard, and start an asynchronous disk flushing task of the old memory table. The memory table switching instruction of the old shard is generated after the first write request is processed and before the second write request is started. The asynchronous disk flushing task is used to asynchronously write the data of the old memory table to the disk file.

[0144] The dual-write module 406 is configured to route the second write request and subsequent write requests to the writable memory table of the old shard and the memory table of the new shard.

[0145] Switching module 408 is configured to, in response to a shard switching event, route a third write request to the memory table of the new shard and release the old shard after the read request on the old shard has been processed.

[0146] Optionally, the dual-write module 406 is further configured as follows:

[0147] Receive a target write request, wherein the target write request is any one of the second write request and subsequent write requests;

[0148] Write the target write request to the writable memory table of the old shard and write the target write request to the memory table of the new shard, and prevent the old shard and the new shard from starting a new flush task.

[0149] Optionally, switching module 408 is further configured as follows:

[0150] Monitor asynchronous disk flushing tasks in older shards;

[0151] When the asynchronous disk flushing task in the old shard is completed, a shard switching event is triggered, and the third write request is routed to the memory table of the new shard, controlling the old shard to enter a read-only state. The third write request is the write request after the shard switching event is triggered.

[0152] Optionally, the database storage architecture is either a read-time merging architecture or a write-time merging architecture. In the case of a write-time merging architecture, each write record is bound to a corresponding logical location identifier, and the disk file includes a logical location hidden column used to mark the logical location identifier of each write record. The database platform also includes a delete request processing module, configured as follows:

[0153] In response to a deletion request, a deletion record is written into the deletion vector, wherein the deletion record includes a logical location identifier of the data to be deleted;

[0154] Receive a data query request for the target data, wherein the data query request carries a logical location identifier to be queried; query the corresponding target physical location based on the logical location identifier to be queried, and the data query request is triggered based on a query task and / or a disk flushing task.

[0155] Optionally, the request processing module is removed and further configured as follows:

[0156] Determine the logical location identifier range for each disk file;

[0157] Based on the logical location identifier range of each disk file, candidate disk files containing the logical location identifier to be queried are determined;

[0158] The logical location identifier to be queried is queried in the candidate disk file to determine the corresponding target physical location.

[0159] Optionally, the request processing module is removed and further configured as follows:

[0160] An approximate member query structure is used to determine whether the logical location identifier to be queried may exist in the candidate disk file;

[0161] If it is determined that it may exist, then read the logical location identifier of each write record of the candidate disk file and find the target physical location corresponding to the logical location identifier to be queried.

[0162] Optionally, the request processing module is removed and further configured as follows:

[0163] When the physical location caching function is enabled, a deletion request is received, the reference physical location of the target data is queried, and a deletion record is written into the deletion vector. The deletion record includes the reference physical location and logical location identifier of the target data.

[0164] The data query request for the target data carries the physical location and logical location identifier of the target data to be queried; the request processing module is removed and further configured as follows:

[0165] If the query based on the physical location to be queried fails, then the query based on the logical location identifier to be queried will be executed to find the corresponding target physical location.

[0166] Optionally, the database platform also includes a configuration module, configured as follows:

[0167] Display the runtime configuration page, which includes startup controls for the physical location caching function;

[0168] In response to the trigger operation of the physical location caching function's startup control, the physical location caching function is started.

[0169] Optionally, the database platform also includes a merge processing module, configured as follows:

[0170] In response to a disk file merge event, determine the source disk files corresponding to the disk file merge event, wherein the disk file merge event includes a source disk file selection event and / or a commit event after the source disk files have been merged;

[0171] Search the memory table for deletion requests associated with each source disk file;

[0172] If the number of deletion requests exceeds the set threshold, the disk file merge event will be canceled.

[0173] This specification provides an embodiment of a database platform, including a creation module, a freeze module, a dual-write module, and a switching module. By actively freezing the old memory table of the old shard during request intervals, asynchronously flushing the old memory table to disk, and initiating dual-write, both the old and new shards simultaneously receive write requests, ensuring no data loss and uninterrupted flow. This ensures the switching process is seamless for the upper layer, achieving uninterrupted write requests during shard switching and enabling simultaneous writing and expansion. It avoids the problems of waiting for long periods to flush the old shard to disk or directly stopping writes, thus providing the ability to dynamically adjust database shards during continuous writing. The entire process requires no downtime and does not interrupt service, supporting elastic scaling of database shards in high-concurrency scenarios. This ensures high availability, improves system elasticity and reliability, and significantly enhances the user experience.

[0174] The above is an illustrative scheme of a database platform according to this embodiment. It should be noted that the technical solution of this database platform and the technical solution of the database resharding method described above belong to the same concept. For details not described in detail in the technical solution of the database platform, please refer to the description of the technical solution of the database resharding method described above.

[0175] Figure 5 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0176] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0177] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0178] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0179] The processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the database resharding method described above.

[0180] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the database resharding method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the database resharding method described above.

[0181] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the database resharding method described above.

[0182] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the database resharding method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the database resharding method described above.

[0183] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the database resharding method described above.

[0184] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the database resharding method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the database resharding method described above.

[0185] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0186] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0187] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0188] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0189] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A database resharding method, comprising: In response to a database resharding request, create the number of new shards corresponding to the resharding request; In response to the memory table switching instruction of the old shard, the old memory table of the old shard is marked as frozen, a writable memory table is created in the old shard, and an asynchronous disk flushing task of the old memory table is started. The memory table switching instruction of the old shard is generated after the first write request is processed and before the second write request is started. The asynchronous disk flushing task is used to asynchronously write the data of the old memory table to the disk file. The second write request and subsequent write requests are routed to the writable memory table of the old shard and the memory table of the new shard. In response to a sharding switch event, a third write request is routed to the memory table of the new shard, and the old shard is released after the read request on the old shard has been processed.

2. The database resharding method according to claim 1, wherein routing the second write request and subsequent write requests to the writable memory table of the old shard and the memory table of the new shard includes: Receive a target write request, wherein the target write request is any one of the second write request and subsequent write requests; The target write request is written to the writable memory table of the old shard and the target write request is written to the memory table of the new shard, and the old shard and the new shard are prevented from starting new disk flushing tasks.

3. The database resharding method according to claim 1, wherein routing the third write request to the memory table of the new shard in response to a sharding switch event includes: Monitor asynchronous disk flushing tasks in the old shards; When the asynchronous disk flushing task in the old shard is completed, the shard switching event is triggered, the third write request is routed to the memory table of the new shard, and the old shard is controlled to enter a read-only state. The third write request is the write request after the shard switching event is triggered.

4. The database resharding method according to any one of claims 1-3, wherein the storage architecture of the database is a read-time merging architecture or a write-time merging architecture; when the storage architecture of the database is the write-time merging architecture, each write record is bound to a corresponding logical location identifier, and the disk file includes a logical location hidden column, the logical location hidden column being used to mark the logical location identifier of each write record; The method further includes: In response to a deletion request, a deletion record is written into the deletion vector, wherein the deletion record includes a logical location identifier of the data to be deleted; Receive a data query request for target data, wherein the data query request carries a logical location identifier to be queried; query the corresponding target physical location based on the logical location identifier to be queried, wherein the data query request is triggered based on a query task and / or a disk flushing task.

5. The database resharding method according to claim 4, wherein querying the corresponding target physical location based on the logical location identifier to be queried includes: Determine the logical location identifier range for each disk file; Based on the logical location identifier range of each disk file, candidate disk files containing the logical location identifier to be queried are determined; The corresponding target physical location is determined by querying the logical location identifier in the candidate disk file.

6. The database resharding method according to claim 5, wherein querying the logical location identifier to be queried in the candidate disk file to determine the corresponding target physical location includes: An approximate member query structure is used to determine whether the logical location identifier to be queried may exist in the candidate disk file; If it is determined that it may exist, then read the logical location identifier of each write record of the candidate disk file and find the target physical location corresponding to the logical location identifier to be queried.

7. The database resharding method according to claim 4, wherein writing a deletion record in the deletion vector in response to a deletion request includes: When the physical location caching function is enabled, a deletion request is received, the reference physical location of the target data is queried, and a deletion record is written into the deletion vector. The deletion record includes the reference physical location and logical location identifier of the target data. The data query request for the target data carries the physical location and logical location identifier of the target data to be queried; After receiving the data query request for the target data, the method further includes: If the query based on the physical location to be queried fails, then the query based on the logical location identifier to be queried is performed to find the corresponding target physical location.

8. The database resharding method according to claim 7, further comprising: Display the runtime configuration page, which includes the startup control for the physical location caching function; In response to the triggering operation of the start control of the physical location caching function, the physical location caching function is started.

9. The database resharding method according to any one of claims 1-3, the method further comprising: In response to a disk file merge event, determine each source disk file corresponding to the disk file merge event, wherein the disk file merge event includes a source disk file selection event and / or a commit event after the source disk files have been merged; Search the memory table for deletion requests associated with each of the source disk files; If the number of deletion requests exceeds a set threshold, the disk file merge event will be canceled.

10. A database platform, comprising: The creation module is configured to create the number of new shards corresponding to the resharding request in response to a database resharding request; The freeze module is configured to, in response to a memory table switching instruction for an old shard, mark the old memory table of the old shard as frozen, create a writable memory table in the old shard, and start an asynchronous disk flushing task for the old memory table. The memory table switching instruction for the old shard is generated after the first write request is processed and before the second write request is started. The asynchronous disk flushing task is used to asynchronously write the data of the old memory table to a disk file. The dual-write module is configured to route the second write request and subsequent write requests to the writable memory table of the old shard and the memory table of the new shard. The switching module is configured to, in response to a shard switching event, route a third write request to the memory table of the new shard and release the old shard after the read request on the old shard has been processed.

11. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the database resharding method according to any one of claims 1-9.

12. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the database resharding method according to any one of claims 1-9.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the database resharding method of any one of claims 1-9.

Images