Log type copy-oriented query method and related equipment
By building an index structure in the log-based replica, the efficient query capability of the log-based replica is realized, which solves the problem that traditional log-based replicas cannot provide query services and improves the resource utilization and read scalability of the database system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-04-14
AI Technical Summary
In traditional database systems, log-based replicas are only used for data synchronization and consistency verification, and cannot provide efficient query services, resulting in limited read scalability, excessive load on the primary replica, and becoming a system bottleneck.
By building a dedicated index structure in a log-based replica, query capabilities can be improved by pre-recording result data of a specified version, allowing queries to be executed directly on the index, while maintaining the low-cost advantages of data synchronization and consistency verification.
Without increasing storage costs, it improves the resource utilization and read load balancing capabilities of the database system, and solves the problems of read scalability and load balancing.
Smart Images
Figure CN121858561A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of database technology, and more particularly to a query method for log-based replicas, a database system, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology
[0002] In related technologies, database systems typically use a sharding mechanism to divide stored data. To ensure high availability and strong consistency of data, a multi-replica mechanism is further adopted. Each data shard generally maintains a primary replica and at least one secondary replica. The primary replica is usually an all-in-one replica, which not only completely saves all the data in the data shard, but also provides read and write services to the outside world, and is responsible for receiving and executing various data operations such as adding, deleting, modifying and querying from the client.
[0003] In some implementations, to reduce storage and maintenance costs, slave replicas in the database system exist only as log replicas. These log replicas mainly participate in the data synchronization and consistency consensus process, and are used to assist the master replica in data recovery or consistency verification. Summary of the Invention
[0004] In view of the above, this specification provides one or more embodiments of a query method for log-type replicas, a database system, an electronic device, a computer-readable storage medium, and a computer program product.
[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a query method oriented towards log-based replicas is proposed, which provides query services through log-based replicas of any data shard in a distributed database, wherein the log-based replica of any data shard is used to record result data corresponding to committed transactions on that data shard; the method includes: Obtain a query request for any of the data shards; Determine the target index structure to respond to the query request, the target index structure being constructed based on the result data of a specified version in the log replica; Based on the query conditions indicated by the query request and the target index structure, the target data is searched in the result data of the specified version; Based on the output conditions indicated by the query request and the target data, the query results are generated and returned.
[0006] According to a second aspect of one or more embodiments of this specification, a database system is provided, comprising at least two database nodes, the database nodes being used to store a primary replica or a secondary replica of a data shard; the secondary replica includes a log-type replica. The database node used to store log-type replicas of data shards is used to execute the query method for log-type replicas described in the first aspect.
[0007] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.
[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.
[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.
[0010] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: The embodiments in this specification pre-build a dedicated index structure (target index structure) for a specific version of the result data that has achieved consistency in the log-type replica. This allows queries to be executed directly based on this index structure. Under the premise of significantly lower storage costs than the universal replica, the traditional log-type replica, which is only used for data synchronization and consistency verification, is transformed into a replica with efficient query capabilities, which significantly improves the resource utilization and read load sharing capabilities of the database system.
[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description
[0012] Figure 1 This is a schematic diagram of the architecture of a database system provided in an exemplary embodiment.
[0013] Figure 2 This is a schematic diagram of a one-master-two-slave replica architecture for a data partition provided in an exemplary embodiment.
[0014] Figure 3 This is a flowchart of a query method for log-based replicas provided in an exemplary embodiment.
[0015] Figure 4 This is a schematic diagram of an index structure provided in an exemplary embodiment.
[0016] Figure 5This is an exemplary embodiment illustrating how the root hash value of an index structure is stored as a consistency verification credential in the log block header of a log-type replica.
[0017] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment. Detailed Implementation
[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0019] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.
[0020] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.
[0021] Please see Figure 1 This specification provides a database system that uses a sharding mechanism to divide stored data. Each shard typically maintains a primary replica and at least one secondary replica. For example... Figure 1 The diagram shows that the stored data is divided into four data shards: data shard P1, data shard P2, data shard P3, and data shard P4. Each data shard maintains one primary replica and two secondary replicas.
[0022] A database system comprises at least two database nodes, each of which can store at least one shard of primary or secondary replicas. For example... Figure 1Database node 1 stores the primary replica of shard P1, the secondary replica of shard P2, the primary replica of shard P3, and the secondary replica of shard P4; database node 2 stores the secondary replica of shard P1, the primary replica of shard P2, the secondary replica of shard P3, and the secondary replica of shard P4; database node 3 stores the secondary replica of shard P1, the secondary replica of shard P2, the secondary replica of shard P3, and the primary replica of shard P4.
[0023] In one implementation of the relevant technology, please refer to Figure 2 The primary replica of a data shard is typically a full-function replica. It not only holds the complete and up-to-date data state of the data shard (full data), but also serves as the primary provider of read and write services for that shard. It is responsible for receiving and processing all types of data operation requests from clients, such as add, delete, modify, and query, and propagating data changes to slave replicas through log synchronization protocols to maintain consistency among replicas.
[0024] Secondary replicas exist solely as log-type replicas. These replicas do not persist complete, directly queryable data pages or data structures; instead, they only store serialized transaction log records (e.g., Redo Log, Write-Ahead Log (WAL) entries, or operation logs) received from the primary replica. The core responsibility of log-type replicas is to participate in data synchronization and distributed consensus processes. Their primary value lies in: ① Assisted data recovery: When the primary replica fails, the system can quickly elect or rebuild a new primary replica with complete data based on the persistent and consensus-based logs in one or more log replicas.
[0025] ② Enhance data durability: By replicating logs to multiple physical nodes (including log replicas), ensure that committed transaction data is not lost even under extreme failure conditions.
[0026] ③ Sharing consensus pressure: In leader-based consensus algorithms, log replicas participate as voting members in leader election and log submission consensus voting, improving the system's fault tolerance.
[0027] However, in this traditional architecture, the functionality of log-based replicas is strictly limited to data synchronization and high availability guarantees. All read operations, especially queries requiring low latency or strong consistency, must be routed to the primary replica or other secondary replicas (if they exist) that maintain the full dataset. This results in the system's read scalability being limited by the number of full replicas, and the primary replica may simultaneously bear a heavy read / write load, becoming a potential bottleneck.
[0028] Based on this, the embodiments in this specification overcome the traditional limitation that log-based replicas cannot serve queries. Without changing their core synchronization responsibilities and low-cost storage advantages, they add efficient query service capabilities. The embodiments in this specification propose a query method for log-based replicas. By pre-constructing a dedicated index structure for a specific version of the result data in the log-based replica, queries can be executed directly on this index. This enables log-based replicas to provide query service capabilities while significantly reducing storage costs compared to full replicas, thereby improving the overall utilization of resources and effectively solving the balance problem between read scalability, load balancing, and resource costs in distributed databases.
[0029] The query method for log-based replicas provided in this specification offers query services through log-based replicas of any data shard in a distributed database. Each log-based replica of a data shard records the result data corresponding to committed transactions on that data shard. Transaction types include query transactions and update transactions; here, committed transactions refer to update transactions. For example, each log-based replica of a data shard records the result data corresponding to multiple versions of that data shard. An index structure can be created for the result data of a specified version to provide query services for that specific version.
[0030] The index structure for a specific version can be built based on the index fields in the result data of that specific version and the storage address of the result data for that specific version. Index fields are those that can be used as query conditions, reflecting the query requirements for that specific version of the result data. Appropriate fields can be selected to build the index based on actual query needs (e.g., frequent queries based on user ID or order time), enabling log-based replicas to efficiently respond to specific types of query requests and transforming general log records into a structured data view that can be efficiently retrieved.
[0031] The target index structure includes a tree structure. Taking a B+ tree structure as an example, each leaf node in the tree structure includes the index field value of each result record in the specified version of the result data, as well as a first pointer to the storage address of that result record. Non-leaf nodes in the tree structure include the index field value used to determine the search path for child nodes, as well as a second pointer to the child nodes. This structure enables equality queries or range queries based on index fields to quickly locate the target result record with approximately logarithmic time complexity, serving as the core data structure guarantee for achieving efficient queries.
[0032] In this design, if the specified version of the result data is stored on disk, then any first pointer points to the storage address of the corresponding result record on disk; alternatively, if the specified version of the result data is read into memory and cached, then any first pointer points to the cache address of the corresponding result record in memory. This design provides storage flexibility: when higher query performance is desired, the specified version of the result data and index can be kept resident in memory; when considering cost and data volume, disk storage can be used, and disk I / O access can be performed directly through pointers, thus achieving a balance between performance and resource consumption.
[0033] Of course, the index structure can also be other types of tree structures, such as B-trees and prefix trees, and this embodiment does not impose any restrictions on this. Different index structures are suitable for different scenarios: B-trees may be more suitable for write-intensive scenarios, while prefix trees are particularly good at handling string prefix matching queries. This flexibility allows the system to select the most suitable index implementation based on the characteristics of the specified version of the result data and the expected query load, further optimizing the query service capabilities of the log replica.
[0034] Through the above mechanism, log replicas are no longer merely passive participants in data synchronization, but have transformed into entities capable of proactively providing data query services for specific consistent versions. A dedicated, read-only index "map" is constructed for a static snapshot of a specific version of the data state within the log replica. Query requests can directly navigate to the data using this "map," completely avoiding the overhead of real-time log replay. This approach maintains the advantages of log replicas—low cost and high synchronization efficiency—while endowing them with powerful read scalability.
[0035] Please see Figure 3 The diagram illustrates a flowchart of a query method for log-based replicas, which can be executed by the database node where the log-based replica resides. The method includes: In S300, a query request for any data shard is obtained.
[0036] For example, the database node hosting the log-based replica can provide a query interface, allowing clients to send query requests to this database node by calling the query interface. Alternatively, the query request can be first received by the database node hosting the primary replica of the data shard, parsed, and then forwarded to the database node hosting the log-based replica for execution, thereby achieving cross-replica query load balancing. This embodiment supports flexible query entry point design, allowing services to be provided directly by the secondary replica, or to be forwarded and distributed with the primary replica, improving the overall availability and query throughput of the system.
[0037] In S302, a target index structure for responding to a query request is determined, which is constructed based on the result data of a specified version in the log-type replica.
[0038] For example, the specified version can be the latest version or the version indicated by the query request.
[0039] In one possible implementation, data sharding employs a multi-version concurrency control mechanism to support transactions; each transaction has a unique transaction identifier, and the types of transactions include query transactions and update transactions, with update transactions including update operations on data shards.
[0040] MVCC (Multi-Version Concurrency Control) is a database concurrency control mechanism that supports the execution of concurrent transactions by maintaining multiple versions of data in the database. Each transaction, when reading data, can see a consistent and visible database state without being affected by other transactions.
[0041] The core idea of MVCC is to create a unique transaction identifier for each transaction and use this identifier to determine transaction visibility. The transaction identifier is an incrementing number. Specifically, when a transaction begins, it is assigned a unique transaction identifier. When reading data, a transaction can only see data committed before its transaction identifier. If another transaction is modifying or deleting a data item, and that transaction's transaction identifier is greater than the current transaction's, then the current transaction will not see the results of these modification or deletion operations.
[0042] For example, the database creates a new version for each modification operation, and each version is associated with the transaction identifier that performed the modification operation. In other words, each data partition corresponds to at least one version, and different versions of the data table serve different ranges of transaction identifiers. The range of transaction identifiers is determined by the transaction identifiers of both the update transaction used to update that version and the update transaction used to update the next version.
[0043] Each operation (create, delete, update, query, etc.) for data partitioning and sharding has a corresponding transaction identifier. Different versions of a data table have two parameters: a minimum transaction identifier and a maximum transaction identifier, which mark the range of transaction identifiers served by that version of the data table. In an example, suppose there is a first version of the result data containing two result records, result record 1 and result record 2. The minimum transaction identifier served by this version of the result data is 100. Now, result record 2 in the first version of the result data is updated, generating the second version of the result data, which includes result record 1 and result record 2'. The update transaction ID is 150, so the maximum transaction identifier of the first version of the result data is set to 149, and the minimum transaction identifier of the second version of the result data is 150. Operations with transaction identifiers in the range [100, 150) will access the first version of the result data; operations with transaction IDs greater than or equal to 150 will access the second version of the result data.
[0044] Different versions of the result data serve different ranges of transaction identifiers; the specific version used to serve the query request can be determined based on the transaction identifier of the query transaction corresponding to the query request and the range of transaction identifiers served by different versions of the result data.
[0045] For example, during the index structure creation phase, the database node where the log-type replica resides maintains the initial version of the result data and its corresponding index structure, which is used to provide consistent and verifiable query capabilities for transactions within a specific range.
[0046] For example, if a database node detects that a log replica has recorded a new version of the result data, it can create a corresponding index structure for the new version of the result data. It should be emphasized that the transaction identifier ranges served by different versions of the result data are strictly distinguished. The transaction identifier ranges served by different versions of the result data are different, and the transaction identifier range corresponding to each version is determined based on the transaction identifiers of the update transaction corresponding to that version and the update transaction corresponding to the next version.
[0047] In this embodiment, by combining the MVCC mechanism with the management of multi-version result data and index structure, the database can maintain consistency and isolation when multiple transactions are concurrent; query transactions always hit the correct version that is visible to them, without locking and blocking update transactions; the index structure is strictly bound to the multi-version data, which can provide efficient and verifiable point lookup or range lookup for each transaction.
[0048] In some embodiments, the result data of a specified version corresponds to one index structure, or the result data of a specified version corresponds to at least two index structures, different index structures correspond to different combinations of index fields, and the combination of index fields includes at least one index field.
[0049] Specifically, at least two index structures are constructed for the specified version of the result data. These include: grouping all index fields of the specified version of the result data according to at least one pre-defined query requirement to obtain a combination of index fields corresponding to each query requirement; and constructing an index structure matching each query requirement based on the combination of index fields corresponding to each query requirement and the storage address of the specified version of the result data. By constructing an efficient index structure that matches the query requirements, log-based replicas gain the ability to quickly locate target data, overcoming the limitation of traditional log-based replicas, which can only be used for operation replay and lack efficient query capabilities.
[0050] For example, a data shard stores a set of order records, including fields such as order number (order_id), user ID (user_id), order date (order_date), and order status (order_status).
[0051] For a query requirement such as "retrieve a specified order by order number", which involves a single-field query, the database node can determine that the index field combination only contains the order number (order_id) and build a single-field index structure based on this index field combination to quickly locate the target order, supporting single-field point queries and single-field range queries.
[0052] Another query requirement, such as "querying all completed orders of a specified user within a certain date range", involves multi-field combination queries. In this case, the database node can determine that the index field combination includes the two fields of user ID (user_id) and order date (order_date), and build a multi-field combination index structure based on this index field combination to support joint filtering of user dimension and time range, and support multi-field point queries and multi-field range queries.
[0053] In the process of providing query services, in a scenario where the result data of a specified version corresponds to an index structure, the unique index structure corresponding to the specified version is the target index structure used to respond to the query request.
[0054] In scenarios where the result data of a specified version corresponds to at least two index structures, the database node can determine the combination of index fields to be retrieved based on the query conditions indicated by the query request, and determine the matching target index structure from at least two index structures corresponding to the specified version based on the combination of index fields to be retrieved.
[0055] In this embodiment, the multi-index support allows for flexible matching of better indexes for different application scenarios (such as single-field point queries, single-field range queries, multi-field point queries, and multi-field range queries).
[0056] In S304, the target data is searched in the result data of the specified version based on the query conditions indicated by the query request and the target index structure.
[0057] For example, a database node can retrieve the target index structure based on the query conditions indicated by the query request to locate the storage address of the target data to be searched, and then read the target data from the specified version of the result data according to the storage address of the target data.
[0058] This embodiment accelerates location through index structure, avoids performing full table scans or sequential traversals on log-based replicas, improves query response speed, and enables low-latency query services even in log-based replicas.
[0059] In S306, the query results are generated and returned based on the output conditions indicated by the query request and the target data.
[0060] In this step, the database node performs post-processing on the target data (such as one or more result records) according to the output conditions indicated by the query request (e.g., columns to be returned, fields to be aggregated, sorting conditions, etc.), such as field truncation, formatting, sorting, or aggregation, to generate the final query result and return the query result to the client.
[0061] The query method for log-based replicas in this specification improves the resource utilization and read load balancing capabilities of the entire database system by transforming traditional log-based replicas, which are only used for data synchronization and consistency verification, into query replicas that also have efficient query capabilities.
[0062] In some embodiments, when the data sharding employs a multi-version concurrency control mechanism to support transactions, to prevent the accumulation of too many historical versions of index structures in the data sharding as transactions continue to execute, thus consuming storage space and increasing maintenance overhead, the database node can execute a version cleanup mechanism in real time or periodically to effectively reclaim old versions of index structures that no longer need to provide services. For example, the database node can detect currently active transactions (including query and update transactions) in the data sharding in real time or according to a preset scheduling period, and determine the smallest transaction identifier among these active transactions (e.g., the smallest transaction ID can represent the earliest still-running transaction start point). Then, it compares the smallest transaction identifier with the range of transaction identifiers served by different versions of the result data to determine the versions of the result data that meet the deletion conditions, and deletes the index structures corresponding to the versions to be deleted; wherein, the deletion conditions include the largest transaction identifier served by the result data of the version to be deleted being less than the smallest transaction identifier.
[0063] For example, if the current data shard contains versions A (service transaction ID range [100, 150)), B (service transaction ID range [150, 200)), and C (service transaction ID range [200, ∞)), and if the database node detects that the smallest active transaction ID currently being executed is 210, it can be determined that the largest transaction IDs (150 and 200) of versions A and B are both less than 210, indicating that they are not visible to the current and subsequent transactions. Therefore, the index structures corresponding to versions A and B can be safely reclaimed, and only the index structure of version C is retained.
[0064] This embodiment effectively reclaims index structures by deleting unused index structures, avoiding the unlimited accumulation of historical versions of index structures, reducing storage costs, effectively reducing long-term occupation of storage space, and preventing dragging down subsequent read and write performance.
[0065] In some embodiments, where the query service strategy is configured to build and maintain the index structure only for the "latest version" result data, there is still a need to clean up the old version index structure. In this scenario, the old version index structure refers to the index structure corresponding to the version that was previously considered the "latest version" but is now outdated, after the data state evolves due to a new update transaction commit and a new "latest version" is generated. To prevent the continuous iteration and accumulation of the index structure as transactions continue to execute, consuming storage space and increasing maintenance overhead, database nodes can establish a corresponding cleanup mechanism. The core principle is that once a new "latest version" is successfully built and its index structure is ready, the index structure of the previous "outdated version" should be safely reclaimed.
[0066] The log-based replica continuously synchronizes its log with the primary replica. Each time it completes the application of a committed update to the log of a transaction, it generates a new data version (the new "latest version"). Database nodes then build their own index structure based on the results of this new version. Once this new index is built and ready for queries, it is established as the "latest version" currently providing query services. For example, a reference pointing to the current "latest version" and its index can be maintained. Whenever a switch to a newer "latest version" is successful, the index structure corresponding to the previously replaced version is marked as an "old index to be deleted" and deleted. This embodiment achieves effective reclamation of index structures by deleting unused index structures. In some embodiments, the index structure also includes data validation functionality to support the verifiability of query results and consistency checks across replicas. See also... Figure 4Taking a Merkle B-tree as an example, the leaf nodes in the index structure include: the index field value of each result record in the result data, a first pointer to the storage address of the result record, and the hash value of the result record (i.e., the hash value obtained by hashing the result record); the non-leaf nodes in the index structure include the minimum / maximum index field value used to find child nodes, a second pointer to the child nodes, and the hash value recorded by the non-leaf nodes in the index structure is obtained by aggregating the hash values recorded by its child nodes.
[0067] Through bottom-up aggregation, the root node of the entire index tree generates a unique root hash value, which fully represents the global state of the indexed data table and its structure at the time of generation. For example... Figure 5 As shown, log entries in a log replica are stored in the form of log blocks. To ensure consistency verification, the root hash value of the index structure is written into the log block header of the log replica as a consistency verification credential.
[0068] When a data shard corresponds to at least two log-type replicas, the root hash value serves as a consistency verification credential and is synchronized to the other log-type replicas corresponding to the data shard. This is used to compare and verify whether the consistency of the data table and index structure among multiple replicas meets the consensus requirements.
[0069] For example, after finding the target data, the database node can also generate a verification path for the target data in the target index structure, and return the verification path and consistency verification credential together with the query results; wherein, the verification path includes: the hash value of the leaf node of the target data in the index structure and the hash values of the sibling nodes at each level on the path from the leaf node to the root node.
[0070] The client receives the query result, the verification path (leaf hash + all sibling hashes), and the root hash value (as a consistency verification credential). The verification steps are as follows: The client first performs a local hash H(data) on the received query result. Then, following the verification path, it concatenates the hash values of sibling nodes from bottom to top, recalculating the parent node hash layer by layer: for example, if there are sibling hashes [H1, H2], it executes H(H(data) || H1|| H2) to obtain the parent node hash. This process continues until the root node is reached, calculating an "expected root hash value." The calculated "expected root hash value" is compared with the root hash synchronized in the received log block header: if they match, it proves that the data does exist in the index structure and has not been tampered with; if they do not match, it proves that the data has been forged or the index structure is at risk of being tampered with. The client does not need to trust the database node providing the service; it can verify that the query result does exist, has not been tampered with, and is consistent with the consensus. Even if an intermediate node or malicious replica attempts to return a forged result, the client can detect it as long as the hash chain does not match.
[0071] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.
[0072] In some embodiments, this specification also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor implements the method described in any one of the above embodiments by executing the executable instructions.
[0073] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 6 As shown, device 600 mainly consists of a communication interface 602, a user interface 604, a processor 606, and a data storage 608. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 610. The communication interface 602 enables device 600 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 602 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 602 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 602 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 602 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.
[0074] User interface 604 includes receiving user input and providing output to the user. Therefore, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 604 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 604 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 600 may support remote access from other devices via communication interface 602 or another physical interface (not shown). User interface 604 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 604 may also be configured as a display device for rendering or displaying text fragments.
[0075] Processor 606 may contain one or more general-purpose processors and / or special-purpose processors.
[0076] Data storage 608 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 606. Data storage 608 may include removable and non-removable components.
[0077] Processor 606 is capable of executing program instructions 618 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 608 to perform the various functions described herein. Data storage 608 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 600, enable device 600 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 618 by processor 606 may result in processor 606 using data 612.
[0078] For example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 600 and one or more applications 620 (e.g., a browser, social application, or game application). Similarly, data 612 may include operating system data 616 and application data 614. Operating system data 616 is primarily accessible to the operating system 622, while application data 614 is primarily accessible to one or more applications 620. Application data 614 may reside in a file system visible or hidden from the user of device 600.
[0079] Application 620 can communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs help application 620 read and / or write application data 614, transmit or receive information via communication interface 602, receive or display information on user interface 604, etc.
[0080] In some terminology, application 620 may be simply referred to as "app". Furthermore, application 620 can be downloaded to device 600 through one or more online app stores or app markets. However, applications can also be installed on device 600 in other ways, such as through a web browser or a physical interface on device 600 (e.g., a USB port).
[0081] In some embodiments, the query apparatus for log-based replicas can be applied to, for example... Figure 6 The device shown implements the technical solution described in this specification. The query apparatus for log-type copies includes: The query request retrieval module is used to retrieve query requests for any data shard.
[0082] The target index structure determination module is used to determine the target index structure to respond to query requests. The target index structure is constructed based on the result data of a specified version in the log replica.
[0083] The target data lookup module is used to find target data in the result data of a specified version based on the query conditions indicated by the query request and the target index structure.
[0084] The query results generation module is used to generate and return query results based on the output conditions indicated by the query request and the target data.
[0085] In one implementation, the target index structure is constructed based on the index field in the result data of the specified version and the storage address of the result data of the specified version.
[0086] The target index structure includes a tree structure; each leaf node in the tree structure includes the index field value of each result record in the specified version of the result data, and a first pointer to the storage address of that result record; the non-leaf nodes in the tree structure include the index field value used to determine the search path of child nodes, and a second pointer to the child nodes.
[0087] In this configuration, the result data of a specified version is stored on the disk, with any first pointer pointing to the storage address of the corresponding result record on the disk; or, the result data of a specified version is read into memory for caching, with any first pointer pointing to the cache address of the corresponding result record in memory.
[0088] In one implementation, the result data of a specified version corresponds to at least two index structures, and different index structures correspond to different combinations of index fields. The combination of index fields includes at least one index field.
[0089] The target index structure determination module is specifically used to determine the combination of index fields to be retrieved based on the query conditions indicated by the query request, and to determine the matching target index structure from at least two index structures corresponding to the specified version based on the combination of index fields to be retrieved.
[0090] In one implementation, an index structure construction module is also included, which is used to group all the index fields included in the result data of a specified version according to at least one preset query requirement, to obtain the index field combination corresponding to each query requirement; and to construct an index structure matching each query requirement based on the index field combination corresponding to each query requirement and the storage address of the result data of the specified version.
[0091] In one implementation, any index structure corresponding to the result data of a specified version includes a tree structure and has data verification functionality. Each leaf node in the index structure includes the hash value of each result record in the result data of the specified version; the hash value recorded by the non-leaf nodes in the index structure is obtained by aggregating the hash values recorded by their child nodes. The root hash value recorded by the root node of the index structure serves as a consistency verification credential and is recorded in the log block header of the log-type replica. The device also includes a verification module for generating a verification path for the target data in the target index structure, and returning the verification path and consistency verification credential together with the query results; wherein, the verification path includes: the hash value of the leaf node of the target data in the target index structure and the hash values of the sibling nodes at each level on the path from the leaf node to the root node.
[0092] In one implementation, the specified version includes the latest version of the data shard; or, the data shard uses a multi-version concurrency control mechanism to support transactions; each transaction has a unique transaction identifier, and the transaction types include query transactions and update transactions; different versions of the result data serve different ranges of transaction identifiers; wherein, the specified version is determined based on the transaction identifier of the query transaction corresponding to the query request, and the range of transaction identifiers served by different versions of the result data.
[0093] In one implementation, the apparatus further includes a deletion module, used to periodically determine the minimum transaction identifier in the current transaction of the data partition when the data sharding adopts a multi-version concurrency control mechanism to support transactions; compare the minimum transaction identifier with the transaction identifier range served by different versions of the result data respectively, determine the version to be deleted that meets the deletion conditions in the different versions of the result data, and delete the index structure corresponding to the version to be deleted; wherein, the deletion conditions include that the maximum transaction identifier served by the result data of the version to be deleted is less than the minimum transaction identifier.
[0094] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0095] Based on the same concept as the above method, this specification also provides a database system including at least two database nodes, wherein the database nodes are used to store primary or secondary copies of data partitions; the secondary copies include log-type copies; wherein the database node used to store the log-type copies is used to execute the above-described query method for log-type copies.
[0096] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0097] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0098] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0099] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.
[0100] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.
[0101] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.
[0102] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.
[0103] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.
[0104] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.
[0105] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.
Claims
1. A query method oriented towards log-based replicas, providing query services through log-based replicas of any data shard in a distributed database, wherein the log-based replica of any data shard is used to record the result data corresponding to committed transactions on that data shard; the method includes: Obtain a query request for any of the data shards; Determine the target index structure to respond to the query request, the target index structure being constructed based on the result data of a specified version in the log replica; Based on the query conditions indicated by the query request and the target index structure, the target data is searched in the result data of the specified version; Based on the output conditions indicated by the query request and the target data, the query results are generated and returned.
2. The method according to claim 1, wherein the target index structure is constructed based on the index field in the specified version of the result data and the storage address of the specified version of the result data; The target index structure includes a tree structure; each leaf node in the tree structure includes an index field value for each result record in the specified version of the result data, and a first pointer for pointing to the storage address of that result record; the non-leaf nodes in the tree structure include an index field value for determining the search path of the child node, and a second pointer for pointing to the child node; in, The specified version of the result data is stored on the disk, and any of the first pointers points to the storage address of the corresponding result record on the disk; or, the specified version of the result data is read into memory for caching, and any of the first pointers points to the cache address of the corresponding result record in memory.
3. The method according to claim 1, wherein the specified version of the result data corresponds to at least two index structures, different index structures correspond to different combinations of index fields, and the combination of index fields includes at least one index field; Determining the target index structure for responding to the query request includes: Based on the query conditions indicated by the query request, determine the combination of index fields to be retrieved, and based on the combination of index fields to be retrieved, determine the matching target index structure from at least two index structures corresponding to the specified version.
4. The method according to claim 3, comprising constructing at least two index structures corresponding to the specified version of the result data, including: Based on at least one preset query requirement, all index fields included in the result data of the specified version are grouped to obtain a combination of index fields corresponding to each query requirement; Based on the combination of index fields corresponding to each query requirement and the storage address of the result data of the specified version, an index structure matching each query requirement is constructed.
5. The method according to any one of claims 1 to 4, wherein any index structure corresponding to the result data of the specified version includes a tree structure and has data verification functionality; Each leaf node in the index structure includes the hash value of each result record in the specified version of the result data; the hash value recorded by the non-leaf nodes in the index structure is obtained by aggregating the hash values recorded by their child nodes; in, The root hash value recorded in the root node of the index structure serves as a consistency verification credential and is recorded in the log block header of the log-type replica. The method further includes: Generate a verification path for the target data in the target index structure, and return the verification path and the consistency verification credential together with the query result; wherein, the verification path includes: the hash value of the leaf node of the target data in the target index structure and the hash values of each sibling node on the path from the leaf node to the root node.
6. The method according to claim 1, wherein the specified version includes the latest version of the data shard; or, The data sharding employs a multi-version concurrency control mechanism to support transactions; each transaction has a unique transaction identifier, and transaction types include query transactions and update transactions; different versions of the result data serve different ranges of transaction identifiers; wherein... The specified version is determined based on the transaction identifier of the query transaction corresponding to the query request, and the range of transaction identifiers served by the result data of different versions.
7. The method according to claim 6, further comprising: When the data sharding uses a multi-version concurrency control mechanism to support transactions, the smallest transaction identifier in the current transaction of the data partition is periodically determined; The minimum transaction identifier is compared with the range of transaction identifiers served by different versions of the result data to determine the version to be deleted that meets the deletion conditions, and the index structure corresponding to the version to be deleted is deleted; wherein, the deletion conditions include the maximum transaction identifier served by the result data of the version to be deleted being less than the minimum transaction identifier.
8. A database system comprising at least two database nodes, wherein the database nodes are used to store primary or secondary replicas of data shards; the secondary replicas include log-type replicas; in, The database node used to store log-based replicas of data shards is used to execute the query method for log-based replicas as described in any one of claims 1 to 7.
9. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 7.
11. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 7.