Distributed columnar database execution plan caching collaborative management method and system

By introducing a shared system table and a cross-node cache synchronization mechanism into the distributed columnar database, the problems of cache silos and low hit rates in the execution plan caching mechanism under the distributed architecture are solved, and the cross-node sharing and reuse of execution plans are realized, thereby improving system performance.

CN123115221APending Publication Date: 2026-11-17JINZHUAN INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611337429.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-31
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In a distributed columnar database architecture with multiple computing nodes, the existing execution plan caching mechanism suffers from cache silos and low hit rates. This results in repetitive execution plan generation work consuming valuable computing resources and hindering the improvement of the overall query performance of the system.

Method used

By using a shared system table as a centralized persistent storage medium for execution plan caching, and in conjunction with a configurable cross-compute node cache synchronization mechanism, a closed-loop cache collaborative management system is constructed, which integrates local generation, system table persistence, global synchronization, and local reuse, thereby enabling cross-node sharing and reuse of execution plans.

Benefits of technology

It effectively breaks the cache silo problem, significantly improves the cache hit rate, reduces redundant execution plan optimization work, reduces CPU resource consumption, and improves the overall query performance of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115221A_ABST
    Figure CN123115221A_ABST
Patent Text Reader

Abstract

This application discloses a method and system for collaborative management of execution plan caching in a distributed columnar database. The method includes: a first computing node receiving a Structured Query Language (SQL) query request and generating a corresponding physical execution plan; the first computing node constructing cache entries based on the physical execution plan and storing the cache entries in its local memory; the first computing node serializing the cache entries and writing them into a shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar database; and a second computing node retrieving the serialized cache entries from the shared system table through a preset synchronization mechanism, deserializing the serialized cache entries, and storing them in its local memory. This ensures that when the second computing node receives a query request identical to the SQL query request, it can retrieve and reuse the physical execution plan from its local memory. This application addresses the technical problem of low hit rate in execution plan caching mechanisms under distributed architectures.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically, to a method and system for collaborative management of distributed columnar database execution plan caching. Background Technology

[0002] This section is intended to provide background or context for the content set forth in the claims or specification, and the content described herein is not acknowledged as prior art simply because it is included in this section.

[0003] With the advent of the big data era, distributed column-oriented databases have been widely applied in data warehousing, business intelligence, and real-time analytics due to their superior performance in massive data analysis scenarios. In a distributed column-oriented database architecture deployed across multiple compute nodes, the system typically consists of multiple Virtual Compute Nodes (VCNs) working collaboratively. Each compute node independently receives and processes Structured Query Language (SQL) query requests initiated by clients. To improve query processing efficiency, each compute node typically maintains an execution plan cache in its local memory, caching the parsed and optimized physical execution plan so that subsequent identical queries can be reused directly, avoiding the overhead of repeated SQL parsing and optimization.

[0004] However, in a distributed architecture deployed across multiple compute nodes, existing execution plan caching mechanisms suffer from core problems such as cache silos and low hit rates. Specifically, each compute node's execution plan cache is maintained only in its local memory, and cached execution plans cannot be shared between different compute nodes. When client query requests are routed to different compute nodes through a load balancing mechanism, even identical SQL queries may require repeated parsing and optimization operations on different compute nodes, each independently generating and caching its own physical execution plan. This leads to a severe waste of cache resources and a low global cache hit rate. The cache silo effect is particularly pronounced in large-scale clusters with a large number of compute nodes, where the repetitive execution plan generation consumes valuable computing resources, hindering further improvements in overall system query performance.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method and system for collaborative management of execution plan caching in a distributed columnar database, which at least solves the technical problem of low hit rate in the execution plan caching mechanism under a distributed architecture.

[0007] According to one aspect of the embodiments of this application, a distributed columnar database execution plan caching collaborative management system is provided, including: multiple computing nodes and a shared system table. The multiple computing nodes include a first computing node and a second computing node. Each computing node has local memory. The shared system table is a persistent storage table accessible to all computing nodes in the distributed columnar database. The first computing node is used to receive SQL query requests and generate corresponding physical execution plans, construct cache entries based on the physical execution plans and store them in local memory, and serialize the cache entries and write them to the shared system table. The second computing node is used to obtain serialized cache entries from the shared system table through a preset synchronization mechanism, deserialize them, and store them in the local memory of the second computing node, so that when the second computing node receives a query request identical to the SQL query request, it can hit and reuse the physical execution plan from its local memory.

[0008] According to another aspect of the embodiments of this application, a method for collaborative management of execution plan caching in a distributed columnar database is also provided, comprising: a first computing node receiving a Structured Query Language (SQL) query request and generating a corresponding physical execution plan; the first computing node constructing cache entries based on the physical execution plan and storing the cache entries in the local memory of the first computing node; the first computing node serializing the cache entries and writing them into a shared system table, wherein the shared system table is a persistent storage table accessible to all computing nodes in the distributed columnar database; and a second computing node obtaining the serialized cache entries from the shared system table through a preset synchronization mechanism, deserializing the serialized cache entries and storing them in the local memory of the second computing node, so that when the second computing node receives a query request identical to the SQL query request, it can hit and reuse the physical execution plan from its local memory.

[0009] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the storage medium including a stored program that executes the above-described method when the program is run.

[0010] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described method through the computer program.

[0011] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the embodiments of the methods described above.

[0012] In this embodiment, a shared system table is used as the centralized persistent storage carrier for the execution plan cache. Combined with a configurable cross-compute node cache synchronization mechanism, a complete cache collaborative management closed loop is constructed, encompassing local generation, system table persistence, global synchronization, and local reuse. This solution effectively breaks the cache silo problem in multi-compute node deployment architectures of distributed column-oriented databases, enabling execution plans generated by any compute node to be shared and reused by all nodes in the cluster. This significantly improves the cache hit rate at the global level, reduces redundant execution plan optimization work, lowers CPU resource consumption, and enhances the overall system query performance. Therefore, it solves the technical problem of low hit rate in execution plan caching mechanisms under distributed architectures. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an optional distributed columnar database execution plan cache collaborative management method according to an embodiment of this application; Figure 2 This is a schematic diagram of a cache synchronization scheme in ZK real-time monitoring mode according to an embodiment of this application; Figure 3 This is a schematic diagram of a cache synchronization scheme under the timed task polling mode according to an embodiment of this application; Figure 4 This is a structural block diagram of a terminal according to an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] According to one aspect of the embodiments of this application, an embodiment of a distributed columnar database execution plan caching collaborative management scheme is provided.

[0017] Optionally, in this embodiment, the distributed columnar database execution plan caching collaborative management method can be applied to the hardware environment constituted by the distributed columnar database execution plan caching collaborative management system of this application. The system includes: multiple computing nodes, including a first computing node and a second computing node, each computing node having local memory; and a shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar database; wherein, the first computing node is used to receive SQL query requests and generate corresponding physical execution plans, construct cache entries based on the physical execution plans and store them in local memory, and serialize the cache entries and write them to the shared system table; the second computing node is used to obtain serialized cache entries from the shared system table through a preset synchronization mechanism, deserialize them and store them in the local memory of the second computing node, so that when the second computing node receives a query request identical to the SQL query request, it can hit and reuse the physical execution plan from its local memory.

[0018] In practical implementation, the distributed columnar database execution plan caching collaborative management system is the system-level implementation plan corresponding to the above method. Multiple computing nodes in the system collectively form a distributed query processing layer, providing a unified SQL query service. Client query requests are distributed to different computing nodes for processing via load balancers or connection pools.

[0019] Each compute node contains the following functional modules: 1) SQL Parsing and Optimization Module: Responsible for parsing, semantic analysis, query rewriting, and cost-based optimization of received SQL query requests to generate the optimal physical execution plan. 2) Local Cache Management Module: Responsible for managing the execution plan cache in local memory, including operations such as cache entry storage, lookup, update, and invalidation cleanup. This module maintains a cache data structure with normalized SQL as the key, supporting efficient key-value lookups. 3) Serialization / Deserialization Module: Responsible for serializing the physical execution plan object in memory into storable JSON format data, and deserializing JSON format data into the physical execution plan object in memory. 4) System Table Interaction Module: Responsible for interacting with the shared system table, including writing locally generated cache entries to the shared system table and querying cache entry data from the shared system table. 5) Synchronization Control Module: Responsible for controlling the triggering and execution process of cache synchronization. According to the configured synchronization mode (real-time monitoring or scheduled task polling), this module initiates synchronization operations through different triggering mechanisms and coordinates other modules to complete data retrieval, deserialization, local cache updates, and other tasks. 6) ZooKeeper Client Module: Responsible for communicating with the ZooKeeper cluster, including registering Watchers to listen for state node changes, reading and listening to configuration items, updating state nodes, and other operations.

[0020] The aforementioned shared system table serves as the system's centralized cache storage layer, deployed on a shared storage infrastructure. It can be a table within a distributed storage system or a system table maintained through a consistency protocol. The shared system table provides highly available and persistent storage services, ensuring that cached data is not lost due to the failure of a single node.

[0021] The aforementioned ZooKeeper cluster serves as the system's coordination service layer, providing configuration management and event notification functions. It stores configuration parameters for cache synchronization mode and uses the Watcher mechanism to implement change notification functionality in real-time monitoring mode.

[0022] During system operation, the computing nodes communicate and collaborate indirectly through a shared system table and ZooKeeper, forming a loosely coupled distributed cache collaborative management system. Execution plans generated by any computing node can be retrieved and reused by other nodes through the shared system table, thereby achieving global cache resource sharing.

[0023] By adopting the above technical solution, the distributed columnar database execution plan caching collaborative management system achieves cross-node collaborative management of execution plan caching through modular design and layered architecture. The system fully utilizes the existing infrastructure of the distributed columnar database (shared system tables and ZooKeeper cluster), and by adding a cache collaborative management function module, it solves the cache silo problem in multi-compute node deployments at a relatively low implementation cost, exhibiting good scalability and maintainability.

[0024] In some embodiments of this application, a collaborative management method for execution plan caching in a distributed columnar database is provided. This method is applied to a distributed columnar database system with multiple computing nodes, including at least one first computing node and at least one second computing node. Each computing node has independent local memory for storing the execution plan cache. The distributed columnar database system also includes a shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar database, used for centrally storing serialized execution plan cache entries.

[0025] Figure 1 This is a flowchart of an optional distributed columnar database execution plan cache collaborative management method according to an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps: In step S102, the first computing node receives a Structured Query Language (SQL) query request and generates the corresponding physical execution plan.

[0026] In step S104, the first compute node constructs cache entries based on the physical execution plan and stores the cache entries in the local memory of the first compute node.

[0027] In step S106, the first compute node serializes the cached entries and writes them into the shared system table, which is a persistent storage table accessible to all compute nodes in the distributed columnar database.

[0028] In step S108, the second computing node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, deserializes the serialized cache entries, and stores them in the local memory of the second computing node, so that when the second computing node receives a query request that is the same as the SQL query request, it can hit and reuse the physical execution plan from the local memory.

[0029] In practical implementation, a distributed column-oriented database refers to a database management system that employs a column-oriented storage engine and is deployed in a distributed architecture. Unlike traditional row-oriented databases, column-oriented databases organize data storage by column, significantly reducing I / O overhead in analytical query scenarios. This makes them particularly suitable for applications such as data warehouses and online analytical processing (OLAP) that require aggregation, statistics, and analysis of large amounts of data. In a multi-compute node deployment architecture, the system increases the number of compute nodes through horizontal scaling to improve overall processing capacity, with each compute node independently handling a portion of the query processing tasks.

[0030] A Virtual Compute Node (VCN) is a computational processing unit in a distributed columnar database. It is responsible for receiving SQL query requests from clients, parsing and optimizing the SQL, generating physical execution plans, scheduling execution tasks, and returning query results. Each VCN is equipped with independent local memory for storing execution plan caches, intermediate result data, session state information, etc. In a typical deployment architecture, VCNs can be deployed on physical servers, virtual machines, or containers, receiving client query requests through load balancing devices or service discovery mechanisms.

[0031] A shared system table is a special type of persistent data table provided by a distributed columnar database, used to store system-level metadata. Unlike ordinary user tables, system tables are typically maintained by the database kernel, stored in a shared storage layer, or have their data consistency guaranteed by a distributed consistency protocol. All compute nodes can access the system table for reading and writing through a unified interface. System tables typically store critical information such as database configuration information, user permission information, table structure metadata, and system running status. This application creatively serializes the execution plan cache and stores it in the shared system table, leveraging the existing globally accessible and persistent characteristics of the system table to achieve centralized management of the execution plan cache.

[0032] The preset synchronization mechanism refers to the triggering method and execution process by which compute nodes retrieve cache entries from the shared system table and synchronize them to local memory. This application provides two configurable synchronization modes: a real-time monitoring mode based on ZooKeeper and a scheduled task polling mode. Each mode has its own characteristics and can be flexibly selected according to the actual business scenario and cluster size. The real-time monitoring mode can achieve near real-time cache synchronization with low latency, but it puts some monitoring pressure on the ZooKeeper cluster; the scheduled task polling mode is simple to implement and has controllable system table access pressure, but its synchronization latency is relatively high.

[0033] By adopting the above technical solution, a complete closed-loop cache collaborative management system is constructed by using a shared system table as the centralized persistent storage carrier for execution plan caching, coupled with a configurable cross-component cache synchronization mechanism. This system enables local generation, system table persistence, global synchronization, and local reuse. This solution effectively breaks the cache silo problem in multi-component node deployment architectures of distributed column-oriented databases, allowing execution plans generated by any compute node to be shared and reused by all nodes in the cluster. This significantly improves the cache hit rate at the global level, reduces redundant execution plan optimization work, lowers CPU resource consumption, and improves the overall system query performance. Therefore, it solves the technical problem of low hit rate in execution plan caching mechanisms under distributed architectures.

[0034] In some alternative implementations of this application, cache entries are stored in key-value pairs, where the key is the normalized SQL statement corresponding to the SQL query request and the value is the physical execution plan.

[0035] In practice, normalized SQL refers to SQL statements obtained by standardizing the original SQL query request. The purpose of normalization is to eliminate differences in SQL statements that do not affect the generation of the execution plan, ensuring that SQL queries with the same semantics but slightly different syntax can match the same cached entry. Specifically, normalization typically includes the following operations: unifying the case of keywords (e.g., unifying "select" to "SELECT"), removing redundant whitespace characters, unifying table aliases, replacing constant values ​​with parameter placeholders, and logically rearranging predicate conditions. Through normalization, the cache hit rate can be effectively improved, preventing cache invalidation due to subtle differences in SQL syntax.

[0036] A physical execution plan (PEP) is a concrete execution plan generated by the database query optimizer after parsing, rewriting, and performing cost-based optimization (CBO) on the SQL statement. Unlike the logical execution plan, which describes "what to do," the PEP details "how to do it," including specific physical operators (such as table scans, index scans, hash joins, sorting, and aggregation), the execution order of operations, data flow, and resource estimates for each operation. Generating a PEP is a CPU-intensive process, especially for complex multi-table join queries, where the optimizer needs to search for the optimal solution among numerous possible execution paths, which is time-consuming. Therefore, caching and reusing PEPs can significantly reduce query processing latency and CPU overhead.

[0037] Key-value storage refers to organizing and managing cached entries in the form of key-value pairs. The key serves as a unique identifier for the cached entry, and the value is the actual content cached. In this scheme, the key is the normalized SQL statement, and the value is the corresponding physical execution plan. When a new query request arrives, the system first normalizes the SQL statement, and then uses the normalized SQL statement as the key to search the local cache. If a match is found, the corresponding physical execution plan is directly retrieved and executed, skipping the parsing and optimization phases. If a match is not found, the complete parsing and optimization process is executed to generate a physical execution plan, which is then used to construct a new cached entry and stored in the local cache.

[0038] By adopting the above technical solution, a key-value pair storage method using normalized SQL statements as cache keys and physical execution plans as cache values ​​ensures that semantically identical SQL queries can accurately match the corresponding execution plan cache, maximizing cache utilization efficiency. Simultaneously, the key-value pair storage structure features high lookup efficiency and simple implementation, meeting the low-latency requirements of query processing for cache access.

[0039] In some alternative implementations of this application, serialization is performed using the lightweight JSON data format.

[0040] In practical implementation, serialization refers to the process of converting objects or data structures in memory into byte streams or strings that can be persistently stored or transmitted over a network. Conversely, deserialization is the process of restoring serialized data back into objects or data structures in memory. In this scheme, since the physical execution plan is a complex data structure stored in the local memory of the compute nodes, it cannot be directly written to the shared system tables for persistent storage. Therefore, it needs to be converted into a storable format through serialization.

[0041] JSON (JavaScript Object Notation) is a lightweight data-interchange format that uses a text format completely independent of programming languages ​​to store and represent data. JSON boasts advantages such as high readability, high parsing efficiency, and strong data structure expressiveness, making it widely used in data storage and network transmission scenarios. In this solution, serializing the physical execution plan into JSON format offers the following advantages: First, JSON is a text format that can be directly stored in text type fields of system tables without requiring special binary storage support; second, JSON has excellent cross-platform compatibility, allowing for correct parsing across different versions and architectures of compute nodes; third, JSON serialization and deserialization are highly efficient, without significantly impacting cache write and read performance; and finally, JSON offers good scalability, maintaining forward compatibility by adding new fields when the execution plan's data structure changes.

[0042] In practical implementation, a physical execution plan typically consists of a series of tree-structured physical operator nodes. Each node contains attributes such as operator type, operation parameters, child node pointers, statistics, and cost estimates. During JSON serialization, the entire execution plan tree is recursively converted into a JSON object structure. Each operator node corresponds to a JSON object, which includes an operator type identifier, parameter fields, and an array of child nodes. After serialization, the generated JSON string can be used as the value part of a cached entry, along with the normalized SQL statement key and other metadata information (such as creation time, version number, expiration flag, etc.), and written to a shared system table.

[0043] The table structure design of a shared system table typically includes the following fields: a cache unique identifier field (such as an auto-incrementing ID or UUID), a normalized SQL field (serving as a cache key), a serialized execution plan field (stores the physical execution plan in JSON format), a creation timestamp field, a last update timestamp field, a version number field, an expiration status field, and a node source field. Among these, the normalized SQL field can be indexed to accelerate queries, the timestamp field is used to support incremental synchronization, and the expiration status field is used to mark the validity of cached entries.

[0044] By employing the above technical solution, the physical execution plan is persisted to a shared system table using the lightweight JSON serialization format, thus achieving a leap from local memory to global persistent storage for execution plan caching. The shared system table, as a centralized storage accessible to all compute nodes, provides the data foundation for cross-node cache sharing and is the core support for breaking down cache silos. The choice of the JSON format achieves a good balance between serialization efficiency, storage overhead, compatibility, and scalability.

[0045] In some optional implementations of this application, the preset synchronization mechanism includes a real-time monitoring mode based on ZooKeeper; in the real-time monitoring mode, after the first compute node writes the cached entries into the shared system table, it updates the state nodes in the ZooKeeper cluster; the second compute node pre-registers to monitor the state nodes in the ZooKeeper cluster, and when the state nodes change, the second compute node receives the change notification and, in response to the change notification, pulls the latest serialized cached entries from the shared system table.

[0046] In practical implementation, ZooKeeper is a distributed, open-source distributed application coordination service. It provides a simple set of primitives that enable distributed applications to implement synchronization services, configuration maintenance, and naming services. In distributed column-oriented database systems, ZooKeeper is typically used as the infrastructure for core functions such as cluster metadata management, distributed locks, leader election, and configuration management. This solution reuses existing ZooKeeper cluster resources and utilizes its Watcher (event listener) mechanism to achieve real-time synchronization notifications for execution plan caching.

[0047] A status node is a ZNode in ZooKeeper used to identify the cache status. In a specific implementation, a dedicated ZNode path can be created for the execution plan cache, for example: / database / plan_cache / version. The value stored in this node can be a globally incrementing version number or a timestamp. Whenever a new cache entry is written to the shared system table or an existing cache entry is changed, the compute node responsible for writing (i.e., the first compute node) updates the value of this status node, incrementing it or updating it to the latest timestamp.

[0048] The Watcher mechanism is one of the core functionalities provided by ZooKeeper. A client can register a Watcher on a ZNode. When the state of that ZNode changes (such as node data modification, node creation, node deletion, or changes to child nodes), the ZooKeeper server sends an event notification to the client that registered the Watcher. Upon receiving the event notification, the client can trigger the corresponding processing logic. The Watcher mechanism is one-time use; that is, after triggering it once, it needs to be re-registered to continue listening for the next change.

[0049] In real-time monitoring mode, the entire cache synchronization process is as follows: 1) Initialization Phase: Upon startup, each compute node in the cluster first loads all valid execution plan cache entries from the shared system table, deserializes them, and stores them in local memory to establish an initial local cache. Simultaneously, each compute node registers a Watcher on the ZooKeeper cache state node to listen for data change events on that node.

[0050] 2) Cache Write Phase: When a compute node (the first compute node) receives a new SQL query request, after parsing and optimizing it to generate a physical execution plan, it first constructs the execution plan into a cache entry and stores it in local memory. Then, it serializes the cache entry into JSON format and writes it to the shared system table. After a successful write, the compute node updates the cache status node in ZooKeeper, updating its value to the new version number or the current timestamp to indicate that the cached data has changed.

[0051] 3) Change notification phase: After detecting a change in the data of the cached state node, the ZooKeeper cluster will send a NodeDataChanged event notification to all compute nodes (secondary compute nodes) that have registered Watchers.

[0052] 4) Pull Synchronization Phase: After receiving the change notification from ZooKeeper, the second compute node triggers the cache synchronization process. This node queries the shared system table to retrieve the latest cached entry data. The query method can be a full query or an incremental query based on timestamps or version numbers. After obtaining the latest serialized cache entry, the second compute node deserializes it into an execution plan object in memory and then updates it to the local memory cache. After the update is complete, the second compute node re-registers a Watcher on the ZooKeeper cache state node to listen for the next change.

[0053] Please refer to Figure 2 The cache synchronization timing in ZK real-time monitoring mode involves the following participants: Client, VCN1 (first compute node), SystemTable (shared system table), ZK (ZooKeeper cluster), and VCN2 (second compute node). The timing sequence is as follows: Step 1: The client sends an SQL query request to VCN1. The client application sends the SQL statement to the distributed columnar database cluster via a database connection, and after load balancing, it is routed to VCN1 for processing.

[0054] Step 2: VCN1 parses the SQL query and generates a physical execution plan. VCN1 performs syntax parsing, semantic analysis, query rewriting, and cost-based optimization on the received SQL statement, ultimately generating the optimal physical execution plan.

[0055] Step 3: VCN1 caches the execution plan in local memory. VCN1 constructs cache entries using normalized SQL as the key and physical execution plan as the value, and stores them in the cache area of ​​local memory for reuse in subsequent identical queries.

[0056] Step 4: VCN1 writes the serialized cache entries to the SystemTable. VCN1 serializes the physical execution plan into JSON format and writes it, along with the normalized SQL key and related metadata, to the shared system table, completing the persistent storage of the cache.

[0057] Step 5: VCN1 updates the ZK state node. After successfully writing to the system table, VCN1 updates the value of the cached state node in the ZooKeeper cluster, incrementing the version number or updating it to the current timestamp to trigger a change notification.

[0058] Step 6: ZK sends a change notification to VCN2. After detecting a state node change, the ZooKeeper cluster sends a NodeDataChanged event notification to VCN2, which has pre-registered a Watcher, to inform it that its cached data has been updated.

[0059] Step 7: VCN2 retrieves the latest cached entries from the SystemTable. Upon receiving the change notification, VCN2 immediately initiates a query to the shared system table to retrieve the latest serialization cached entry data.

[0060] Step 8: VCN2 deserializes and stores the data in local memory. VCN2 deserializes the obtained JSON-formatted cache entries, restoring them to a physical execution plan object in memory, and then stores it in the local memory cache. If a cache entry with the same key already exists locally, it is overwritten and updated.

[0061] Step 9: The Client sends the same SQL query request to VCN2. When subsequent identical SQL query requests are routed to VCN2 via load balancing, VCN2 looks up the corresponding entry in its local memory cache.

[0062] Step 10: VCN2 hits the local cache and reuses the execution plan. Since the execution plan has already been cached in local memory through the synchronization mechanism, VCN2 directly hits the cache, skips the parsing and optimization stage, directly executes the physical execution plan, and returns the result.

[0063] By employing the above technical solution, near real-time synchronization of the execution plan cache is achieved based on ZooKeeper's real-time monitoring mode. When a compute node generates a new execution plan and writes it to the shared system table, other compute nodes can detect the change through ZooKeeper's Watcher mechanism within a very short time and proactively pull the latest cached data for synchronization. This mode has the advantages of low synchronization latency and good cache consistency, making it suitable for deployment scenarios with high requirements for cache real-time performance and moderate cluster size. Furthermore, this solution fully reuses the existing ZooKeeper infrastructure of the distributed database, eliminating the need to introduce additional components.

[0064] In some optional implementations of this application, the preset synchronization mechanism includes a scheduled task polling mode. In the scheduled task polling mode, the second computing node starts a background scheduled task to periodically query the shared system table according to a preset time frequency to obtain serialized cache entries. After the second computing node deserializes the serialized cache entries it has queried, it compares the key value with the existing cache entries in the local memory of the second computing node, and performs operations such as adding cache entries, updating changed cache entries, or cleaning up expired cache entries based on the comparison results.

[0065] In practical implementation, the scheduled task polling mode refers to the compute node periodically and proactively querying cached entries in the shared system table at pre-set time intervals through a scheduled task running in the background. The query results are then compared with the local cache, and corresponding synchronization operations are performed based on the comparison results. Unlike the real-time monitoring mode, which relies on ZooKeeper event notifications, the scheduled task polling mode actively retrieves data using a "pull" approach, making its implementation mechanism simpler and more direct.

[0066] Background scheduled tasks are independent threads or coroutines running in the background of each compute node, responsible for periodically executing cache synchronization logic. The trigger frequency of these tasks is controlled by configuration parameters and can be flexibly adjusted according to business needs. For example, it can be set to execute every 30 seconds or every 5 minutes. A higher synchronization frequency can reduce cache synchronization latency but increases access pressure on shared system tables; conversely, a lower synchronization frequency can reduce access pressure on system tables, but cache synchronization latency is relatively higher. In actual deployment, a trade-off needs to be struck based on cluster size, query characteristics, and performance requirements.

[0067] Key value comparison involves comparing cached entries retrieved from the shared system tables with existing cached entries in local memory to identify which are new, modified, or expired cached entries, and then performing different synchronization operations accordingly. The specific comparison logic is as follows: 1) Adding a new cache entry: If a cache entry exists in the shared system table but the corresponding key does not exist in the local cache, it means that this is a newly generated cache entry, which needs to be deserialized and added to the local cache.

[0068] 2) Update changed cache entries: If there are cache entries with the same key in the shared system table and the local cache, but their contents (or version number, timestamp) are different, it means that the cache entry has been updated and the latest version in the shared system table needs to be used to overwrite the old version in the local cache.

[0069] 3) Clean up expired cache entries: If a cache entry exists in the local cache, and the corresponding entry in the shared system table has been marked as expired or deleted, the corresponding entry in the local cache needs to be cleaned up to avoid using an expired execution plan.

[0070] Please refer to Figure 3 The cache synchronization timing in the scheduled task polling mode involves the following participants: VCN1 (first compute node), SystemTable (shared system table), and VCN2 (second compute node). The timing sequence is as follows: Step 1: VCN1 writes the serialized cached entries to the SystemTable. Similar to the real-time monitoring mode, after VCN1 generates a new physical execution plan, it serializes it into JSON format and writes it to the shared system table for persistent storage.

[0071] Step 2: VCN2 starts a background scheduled task. When VCN2 starts, it initializes a background scheduled task thread, which is periodically woken up to execute cache synchronization logic at preset time intervals (such as every 30 seconds).

[0072] Step 3: VCN2 periodically queries the SystemTable to retrieve cached entries. When the scheduled task is triggered, VCN2 sends a query request to the shared system table to retrieve all currently valid serialized cached entry data.

[0073] Step 4: VCN2 performs key value comparison. VCN2 compares the list of cached entries obtained from the system table with the existing cached entries in local memory one by one, and identifies new, modified and invalid entries by matching key values.

[0074] Step 5a: Synchronize and add cached entries. For entries that exist in the system table but not in the local cache, VCN2 deserializes them and adds them to the local memory cache.

[0075] Step 5b: Update modified cache entries. For entries with the same key but different content or version, VCN2 uses the latest version from the system table to overwrite and update the local cache.

[0076] Step 5c: Clean up expired cache entries. For entries that exist in the local cache but have been marked as expired in the system table, VCN2 removes them from the local memory cache.

[0077] Step 6: VCN2 waits for the next scheduled task to be triggered. After completing this synchronization, the scheduled task thread enters a waiting state until the next synchronization cycle arrives and the synchronization process is executed again.

[0078] By employing the above technical solution, the scheduled task polling mode provides a simple and stable cache synchronization mechanism. This mode does not rely on ZooKeeper's event notification mechanism; it completes cache synchronization solely through periodic proactive queries, reducing dependence on and pressure on the ZooKeeper cluster. Furthermore, the execution frequency of the scheduled tasks can be flexibly configured according to actual needs, adapting to cluster environments of different sizes and performance requirements. For large-scale clusters or scenarios with relatively relaxed real-time cache requirements, the scheduled task polling mode is a choice that combines reliability and economy.

[0079] In some optional implementations of this application, the preset synchronization mechanism switches modes based on ZooKeeper configuration items, including real-time monitoring mode and scheduled task polling mode.

[0080] In practical implementation, ZooKeeper configuration items refer to configuration parameters stored in the ZooKeeper cluster, used to control the operating mode of the cache synchronization mechanism. Managing configuration through ZooKeeper offers the following advantages: First, configuration data is centrally stored in ZooKeeper, with all compute nodes sharing the same configuration, ensuring configuration consistency; second, utilizing ZooKeeper's Watcher mechanism, compute nodes can detect configuration changes in real time and dynamically switch operating modes without restarting the service; third, ZooKeeper provides persistent storage and high availability guarantees for configuration data.

[0081] In the implementation, a dedicated configuration node can be created in ZooKeeper, for example, at the path: / database / plan_cache / sync_mode. The value stored in this node is the current synchronization mode configuration, which can be "realtime" (real-time monitoring mode) or "polling" (scheduled polling mode). Additionally, other mode-related parameters, such as the polling interval in the scheduled polling mode, can be stored in the same configuration path.

[0082] Upon startup, each compute node first reads the current synchronization mode configuration from the ZooKeeper configuration node and then initiates the corresponding synchronization mechanism based on the configuration values. Simultaneously, each compute node registers a Watcher on the configuration node to listen for configuration change events. When an administrator modifies the synchronization mode configuration in ZooKeeper using the configuration management tool, all compute nodes receive a configuration change notification and then dynamically switch synchronization modes based on the new configuration values. If you switch from real-time monitoring mode to scheduled task polling mode, then the Watcher of the ZooKeeper state node is turned off, and the scheduled task thread is started to begin polling.

[0083] If switching from scheduled task polling mode to real-time listening mode, the scheduled task thread is stopped, and a Watcher is registered on the ZooKeeper state node to enter the listening state.

[0084] The dual-mode configurable switching mechanism also supports finer-grained configuration strategies. For example, different synchronization modes can be configured based on the role or group of compute nodes. Some critical nodes can use real-time monitoring mode to achieve lower synchronization latency, while other nodes can use scheduled task polling mode to reduce system table access pressure. Furthermore, adaptive mode switching can be implemented. For instance, when excessive system table access pressure is detected, it automatically switches from real-time monitoring mode to scheduled task polling mode to reduce pressure; when cache change frequency is low, it automatically switches from scheduled task polling mode to real-time monitoring mode to achieve better real-time performance.

[0085] By adopting the above technical solution and utilizing a dual-mode configurable switching mechanism based on ZooKeeper configuration items, this solution offers significant deployment flexibility. Operations personnel can flexibly select and dynamically adjust the cache synchronization mode according to the actual operating status of the cluster, business load characteristics, and performance requirements, without modifying code or restarting services. This configurable architecture design enables this solution to be widely adaptable to distributed columnar database deployment scenarios of various sizes and characteristics, demonstrating strong practical value and adaptability.

[0086] In some optional embodiments of this application, when the second computing node deserializes the serialized cache entry and stores it in local memory, if a cache entry with the same key already exists in local memory, the deserialized cache entry is used to overwrite and update the cache entry in local memory.

[0087] In practice, overwriting update refers to replacing the old cached entry in the local cache with the newly synchronized cached entry when a cached entry synchronized from a shared system table has the same key value as an existing cached entry in the local cache. The execution plan cache needs to support overwriting updates because changes in the table structure, statistics, and data distribution in the database can alter the optimal execution plan for the same SQL query. When the execution plan is re-optimized and generated for these reasons, the old plan in the cache needs to be updated promptly to ensure that subsequent queries can use the latest optimal execution plan.

[0088] In the implementation of overwrite updates, atomicity and consistency issues need to be considered. Since the execution plan is a relatively complex data structure, the update process may take some time. If a query request is accessing the cached entry during the update process, incomplete or inconsistent data may be read. To avoid this, one of the following two strategies is typically used: 1) Copy-On-Write strategy: When updating a cached entry, a new, complete cached entry object is first constructed, and then the old entry is replaced with the new entry through an atomic pointer substitution operation. This approach ensures that read operations always see a complete and consistent cached entry, without any intermediate states.

[0089] 2) Read-write lock strategy: Read-write locks are used to protect access to cache entries. Read operations acquire read locks, and update operations acquire write locks. Due to the read-heavy and write-light nature of the execution plan cache, read-write locks can provide better concurrency performance.

[0090] Furthermore, a version number or timestamp verification mechanism can be introduced during overwrite updates. Each cached entry carries a version number or update timestamp. Before performing an overwrite update, the version numbers of the entry to be written and the local entry are compared. The update operation is only performed if the version number of the entry to be written is greater than that of the local entry, thus avoiding the problem of old versions overwriting new versions due to network latency or out-of-order delivery.

[0091] Furthermore, when a cached entry in the shared system table is marked as invalid, the second compute node obtains the invalidation mark through a preset synchronization mechanism and cleans up the corresponding cached entry in the local memory of the second compute node.

[0092] Cache invalidation refers to the situation where, when the underlying data metadata (such as table structure, indexes, statistics, etc.) changes, or when the database objects corresponding to the execution plan (such as tables, views) are deleted or modified, the original execution plan may no longer be applicable. In such cases, the corresponding cached entries need to be marked as invalid to prevent subsequent queries from using incorrect or inefficient execution plans. Common triggers for cache invalidation include: DDL operations (such as ALTER TABLE, CREATE INDEX, DROP TABLE, etc.), updates to statistics, and execution plan invalidation caused by large-scale data changes.

[0093] In this scheme, cache invalidation operations are also propagated through a shared system table. When a compute node detects that a cached execution plan or batch of execution plans needs to be invalidated, it marks the corresponding cached entry as invalid in the shared system table (by setting the invalidation flag field to true or updating the invalidation timestamp field), instead of deleting it directly. The advantages of this approach are twofold: firstly, the invalidation flag can be detected by other compute nodes through synchronization mechanisms, allowing them to clean up their local caches; secondly, retaining invalidation records prevents invalidation operations from being missed in certain abnormal situations.

[0094] When the second compute node performs cache synchronization, it checks the invalidation flags of entries in the shared system table. For cache entries marked as invalid, if a corresponding entry exists in the local cache, it removes it from the local cache. For cache entries that have been invalid for a certain period of time, a background cleanup task can periodically physically delete them from the shared system table to free up storage space.

[0095] By employing the above technical solution, through the combined use of an overwrite update mechanism and an invalidation cleanup mechanism, this approach ensures the accuracy and timeliness of execution plans in the local cache of each computing node. Overwrite updates guarantee that execution plans are updated to the latest version in a timely manner, while invalidation cleanup ensures that inapplicable execution plans are removed promptly. Together, they maintain the consistency and correctness of cached data, providing a reliable guarantee for the collaborative management of execution plan caches in a distributed environment.

[0096] In some optional embodiments of this application, the second computing node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, including: the second computing node recording the timestamp of the most recent successful synchronization operation; when the current synchronization operation is triggered, the second computing node initiates an incremental query to the shared system table, which is used to obtain cache entries in the shared system table that have changed after the timestamp; the second computing node only deserializes the obtained incrementally changed cache entries, updates them to the local memory of the second computing node, and updates the timestamp to the time of the current synchronization operation.

[0097] In practice, incremental synchronization is the opposite of full synchronization. Full synchronization refers to querying all cached entries from the shared system table and then comparing them with the local cache during each synchronization. The advantage of full synchronization is its simplicity and low risk of data loss; the disadvantage is that when the number of cached entries is large, each synchronization requires the transmission of a large amount of data, putting significant pressure on network bandwidth and the query performance of the system table. This is especially true when there are many compute nodes, where full synchronization may lead to excessive access pressure on the system table.

[0098] Incremental synchronization only queries cached entries that have changed since the last synchronization (including newly added, updated, and expired entries), significantly reducing data transfer volume and query overhead. The key to implementing incremental synchronization is maintaining a synchronization watermark (i.e., the timestamp of the most recent successful synchronization) and performing incremental queries based on this watermark.

[0099] A timestamp is a time stamp used to identify when a cached entry was modified. In the shared system table, each cached entry contains an update timestamp field, recording the time when the entry was last created or modified. The second compute node maintains a "recent synchronization timestamp" variable locally, recording the time when the last successful synchronization operation was performed. Each time synchronization is performed, the compute node uses this timestamp as a query condition to retrieve all cached entries from the shared system table whose update timestamps are greater than this value. These entries represent the incremental data that has changed since the last synchronization.

[0100] The specific execution process of incremental synchronization is as follows: 1) Obtaining the local timestamp: At the start of each synchronization operation, the second compute node first reads the most recent synchronization timestamp from its local records. Initially (such as the first synchronization when the node starts up), this timestamp can be set to 0 or a very early time to ensure that all cached entries are obtained during the first synchronization.

[0101] 2) Initiate an incremental query: The second compute node initiates an incremental query to the shared system table using the most recent synchronization timestamp as a parameter to retrieve all cached entries whose update timestamps are later than that timestamp. The query results include all newly added, updated, and expired entries.

[0102] 3) Processing incremental data: The second computing node traverses the incremental entries queried and processes them one by one: For newly added or updated entries, they are deserialized and stored in the local cache (if they already exist, they are overwritten and updated); for entries marked as invalid, the corresponding entries are deleted from the local cache.

[0103] 4) Update timestamp: After the incremental data processing is completed, the second computing node updates the local most recent synchronization timestamp to the time of this synchronization operation (or the latest update timestamp in the incremental data), as the starting point for the next incremental synchronization.

[0104] Incremental synchronization can be applied to both real-time monitoring mode and scheduled task polling mode. In real-time monitoring mode, when a compute node receives a ZK change notification, it uses incremental queries to retrieve the changed data from the system tables. In scheduled task polling mode, incremental queries are also used each time a scheduled task is triggered. In both modes, incremental synchronization significantly reduces data transfer volume, improves synchronization efficiency, and alleviates the access pressure on shared system tables.

[0105] In practical applications, to avoid missing some change records in incremental queries due to timestamp accuracy issues or network latency, a "timestamp rollback" strategy can be adopted. This involves rolling back the query's starting timestamp by a small time window (e.g., 1 second) during each incremental query to ensure that no change records near the boundary are missed. Additionally, a full synchronization can be performed periodically as a check to ensure eventual consistency between the local cache and the system table data.

[0106] By adopting the above technical solution and optimizing the incremental synchronization mechanism based on timestamps, the amount of data transmission and the query overhead of shared system tables during cache synchronization are significantly reduced, thereby improving synchronization efficiency and minimizing the impact on normal system operations. The combined use of incremental and full synchronization ensures data consistency while achieving efficient cross-node cache synchronization, making it particularly suitable for distributed cluster scenarios with a large number of cache entries and a large node scale.

[0107] In summary, this application has the following beneficial effects: 1) Significantly improves global cache hit rate. By persisting the execution plan cache to a shared system table accessible to all compute nodes, and in conjunction with a cross-node cache synchronization mechanism, the cache silos between compute nodes are effectively broken down. This allows the execution plan generated by any compute node to be shared and reused by all compute nodes in the cluster, significantly improving the overall global cache hit rate of the distributed cluster and reducing the redundant execution plan generation work.

[0108] 2) Significantly reduce CPU overhead and improve system throughput. Because execution plans can be shared and reused across multiple computing nodes, the same SQL query is avoided from being repeatedly parsed, rewritten, and optimized on different computing nodes, which is a CPU-intensive operation. This effectively reduces the CPU resource consumption of each computing node, allowing more computing resources to be used for actual data query processing, thereby improving the overall query throughput and response performance of the system.

[0109] 3) Flexible architectural adaptability. This application provides two configurable cache synchronization mechanisms: real-time monitoring mode and scheduled task polling mode. These can be flexibly switched via ZooKeeper configuration options, adapting to distributed cluster deployment scenarios of different sizes and performance requirements. For scenarios with high real-time requirements, the ZK real-time monitoring mode can be selected to achieve near real-time cache synchronization; for scenarios with a large number of nodes and relatively relaxed consistency and real-time requirements, the scheduled task polling mode can be selected to reduce the access pressure on system tables.

[0110] 4) Lightweight implementation with no additional external dependencies. This solution fully reuses the existing system table infrastructure and ZooKeeper cluster resources of the distributed columnar database. It persists the execution plan to the shared system table through serialization / deserialization mechanisms. There is no need to introduce additional centralized caching middleware such as Redis, nor is there any need to make major modifications to the existing database architecture. It has low implementation cost, simple deployment, and is easy to implement in existing distributed columnar database systems.

[0111] 5) Strong consistency guarantee. Through persistent storage based on shared system tables and a configurable synchronization mechanism, this solution can achieve eventual consistency of the execution plan cache. Whether in real-time monitoring mode or scheduled task polling mode, the local cache of each compute node will eventually be consistent with the global cache in the shared system tables, ensuring the accuracy and reliability of cached data and avoiding query result errors or performance fluctuations caused by cache inconsistency.

[0112] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0114] According to another aspect of the embodiments of this application, a distributed column-oriented database execution plan cache collaborative management device for implementing the above-described distributed column-oriented database execution plan cache collaborative management method is also provided. The device may include: The generation module is used to instruct the first computing node to receive a Structured Query Language (SQL) query request and generate the corresponding physical execution plan. A storage module is configured to instruct the first computing node to construct cache entries based on the physical execution plan and store the cache entries in the local memory of the first computing node; The write module is used to instruct the first computing node to serialize the cached entries and write them into the shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar storage database. The reuse module is used to instruct the second computing node to obtain serialized cache entries from the shared system table through a preset synchronization mechanism, and to deserialize the serialized cache entries and store them in the local memory of the second computing node, so that when the second computing node receives a query request that is the same as the SQL query request, it can hit and reuse the physical execution plan from its local memory.

[0115] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in a corresponding hardware environment, and can be implemented through software or hardware, wherein the hardware environment includes a network environment.

[0116] According to another aspect of the embodiments of this application, a server or terminal for implementing the above-described distributed columnar database execution plan cache collaborative management method is also provided.

[0117] Figure 4 This is a structural block diagram of a terminal according to an embodiment of this application, such as... Figure 4 As shown, the terminal may include: one or more (only one is shown in the figure) processors 401, memory 403, and transmission devices 405, such as... Figure 4 As shown, the terminal may also include input / output devices 407.

[0118] The memory 403 can be used to store software programs and modules, such as the program instructions / modules corresponding to the distributed columnar database execution plan cache collaborative management method and apparatus in this embodiment. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 403, thereby realizing the aforementioned distributed columnar database execution plan cache collaborative management method. The memory 403 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 403 may further include memory remotely located relative to the processor 401, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0119] The aforementioned transmission device 405 is used to receive or send data via a network, and can also be used for data transfer between the processor and memory. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 405 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 405 is a radio frequency (RF) module used for wireless communication with the Internet.

[0120] Specifically, memory 403 is used to store application programs.

[0121] The processor 401 can invoke the application program stored in the memory 403 via the transmission device 405 to perform the following steps: The first computing node receives SQL query requests and generates the corresponding physical execution plan; The first computing node constructs cache entries based on the physical execution plan and stores the cache entries in the local memory of the first computing node; After the first computing node serializes the cached entries, it writes them into a shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar database. The second computing node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, deserializes the serialized cache entries and stores them in the local memory of the second computing node, so that when the second computing node receives a query request that is the same as the SQL query request, it can hit and reuse the physical execution plan from its local memory.

[0122] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0123] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. The terminal can be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, the terminal may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0124] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0125] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to execute program code for a distributed columnar database execution plan cache collaborative management method.

[0126] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0127] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: The first computing node receives SQL query requests and generates the corresponding physical execution plan; The first computing node constructs cache entries based on the physical execution plan and stores the cache entries in the local memory of the first computing node; After the first computing node serializes the cached entries, it writes them into a shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar database. The second computing node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, deserializes the serialized cache entries and stores them in the local memory of the second computing node, so that when the second computing node receives a query request that is the same as the SQL query request, it can hit and reuse the physical execution plan from its local memory.

[0128] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0129] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0130] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0131] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0132] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A distributed columnar database execution plan cache co-management system, comprising: The system includes: Multiple computing nodes and a shared system table, wherein the multiple computing nodes include a first computing node and a second computing node, each computing node has local memory, and the shared system table is a persistent storage table accessible to all computing nodes in the distributed columnar database; The first computing node is used to receive SQL query requests and generate corresponding physical execution plans, construct cache entries based on the physical execution plans and store them in local memory, and serialize the cache entries and write them into the shared system table; the second computing node is used to obtain serialized cache entries from the shared system table through a preset synchronization mechanism, deserialize them and store them in the local memory of the second computing node, so that when the second computing node receives a query request that is the same as the SQL query request, it can hit and reuse the physical execution plan from its local memory. 2.A method for cooperative management of distributed columnar database execution plan cache, characterized in that, include: The first computing node receives SQL query requests and generates the corresponding physical execution plan; The first computing node constructs cache entries based on the physical execution plan and stores the cache entries in the local memory of the first computing node; After the first computing node serializes the cached entries, it writes them into a shared system table, which is a persistent storage table accessible to all computing nodes in the distributed columnar database. The second computing node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, deserializes the serialized cache entries and stores them in the local memory of the second computing node, so that when the second computing node receives a query request that is the same as the SQL query request, it can hit and reuse the physical execution plan from its local memory.

3. The method of claim 2, wherein, The first compute node constructs cache entries based on the physical execution plan, including: The first computing node constructs the cache entry in key-value form, using the normalized SQL statement corresponding to the SQL query request as the key and the physical execution plan as the value.

4. The method according to claim 2, characterized in that... Before the second computing node obtains the serialized cache entry from the shared system table through a preset synchronization mechanism, the method further includes: the second computing node pre-registers to listen to the state node in the ZooKeeper cluster, thereby entering a ZooKeeper-based real-time listening mode, wherein the preset synchronization mechanism includes a ZooKeeper-based real-time listening mode. The second compute node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, including: in the real-time monitoring mode, after the first compute node writes the cache entries into the shared system table, it updates the status node in the ZooKeeper cluster; When the state node changes, the second computing node receives a change notification and, in response to the change notification, pulls the latest serialized cache entry from the shared system table.

5. The method of claim 2, wherein, The preset synchronization mechanism includes a timed task polling mode. The second computing node obtains serialized cache entries from the shared system table through the preset synchronization mechanism, deserializes the serialized cache entries, and stores them in the local memory of the second computing node, including: In the scheduled task polling mode, the second computing node starts a background scheduled task to periodically query the shared system table according to a preset time frequency to obtain serialized cache entries; The second computing node deserializes the serialized cache entries it finds and compares them with the existing cache entries in the local memory of the second computing node. Based on the comparison results, it adds cache entries, updates changed cache entries, or cleans up expired cache entries.

6. The method of claim 2, wherein, The method further includes: For any computing node, the synchronization mechanism can be switched between real-time monitoring mode and scheduled task polling mode based on ZooKeeper configuration options.

7. The method of claim 2, wherein, After deserializing the serialized cache entries and storing them in the local memory of the second computing node, the method further includes: When a cached entry in the shared system table is marked as invalid, the second compute node obtains the invalidation mark through the preset synchronization mechanism and clears the corresponding cached entry in the local memory of the second compute node.

8. The method of claim 2, wherein, The second computing node obtains serialized cache entries from the shared system table through a preset synchronization mechanism, including: The second computing node records the timestamp of the most recent successful synchronization operation; When this synchronization operation is triggered, the second computing node initiates an incremental query to the shared system table. The incremental query is used to obtain cache entries in the shared system table that have changed after the timestamp. The second computing node only deserializes the cached entries of the incremental changes it obtains, updates them to the local memory of the second computing node, and updates the timestamp to the time of this synchronization operation.

9. A computer readable storage medium, characterized in that, The storage medium includes a stored program, wherein the program executes the method described in any one of claims 2 to 8 when it is run.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the method described in any one of claims 2 to 8 through the computer program.