Caching apparatus and method for multi-version data based on graph database

Abstract

The application provides a cache device and method for multi-version data based on a graph database, and relates to the technical field of data processing. The device comprises: a data storage module which continuously stores relationship data corresponding to edges and nodes based on an append-only mode, and determines a physical offset value of a scan end position corresponding to a read request; a statistical analysis module which pre-adjusts preset default operators according to actual business requirements and collects statistical data; a cache operator selection module which determines a target operator from the pre-adjusted default operators according to the statistical data, and opens or closes the cache function of the target operator; a cache module which independently saves, version-maintains and eliminates each node of the target operator to generate overall cache information; and a query execution module which determines the optimal version cache corresponding to a target node in the target operator from the overall cache information based on the relationship data and the read request, adjusts a preliminary execution plan based on the optimal version cache and the physical offset value, and executes a corresponding query action.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, for example, to a caching device and method for multi-version data based on a graph database. Background Art

[0002] A graph database is a database management system that uses nodes and edges as basic storage units, directly storing and processing relational data between entities. Graph databases, whose theoretical foundations are derived from graph theory, are particularly well-suited for processing datasets with complex relationships and supporting efficient graph traversal operations. With the rapid development of information technology, the amount of data generated and stored has increased exponentially. Furthermore, as the complexity and scale of graph data continue to grow, the computational complexity of queries has significantly increased. Graph database queries typically require extensive node traversals and edge scans, which are costly operations, but the resulting result sets may not be large. For example, in social network analysis, while traversing numerous user relationships may ultimately require only the few nodes closest to a specific user that meet the criteria. Furthermore, in graph scenarios, the results of different queries often overlap. This characteristic makes caching results particularly valuable. By caching query results that are computationally expensive, have low data volumes, and are likely to be reused, or by caching interim results for complex queries, subsequent queries, once cached, can not only reduce data accesses but also avoid repeated, intensive computations, significantly improving query performance. In distributed environments, data transmission between nodes can introduce additional query latency and processor usage. A reasonable caching strategy can also significantly reduce the network communication overhead in subsequent queries.

[0003] In related technologies, read-only query results are generally cached, that is, the return of the entire query request is cached. However, in subsequent use, only exactly the same query can hit the cached read-only query results, and once the data is updated, the cache may become invalid, and the query needs to be re-performed according to a series of calculation processes, resulting in low query efficiency. Summary of the Invention

[0004] The present application aims to provide a caching device and method for multi-version data based on a graph database.

[0005] According to one aspect of the present application, a multi-version data caching device based on a graph database is proposed, comprising:

[0006] A data storage module is used to continuously store the relationship data corresponding to the edges and nodes in an append-only manner, and determine the physical offset value of the scan end position corresponding to the read request based on the read request and the relationship data;

[0007] The statistical analysis module is used to pre-adjust the preset default operators according to actual business needs and collect statistical data;

[0008] The cache operator selection module is configured to determine a target operator from the pre-adjusted default operator according to the statistical data, and to start or stop the cache function of the target operator, wherein the target operator is an operator that supports caching intermediate results.

[0009] The cache module is configured to independently save, maintain versions, and discard each node of the target operator to generate overall cache information.

[0010] The query execution module is configured to determine an optimal version cache of the target node from the overall cache information based on the relational data and the read request, to adjust a preset preliminary execution plan based on the optimal version cache and the physical offset value, and to perform a corresponding query action.

[0011] According to an aspect of the present application, a cache method for multi-version data based on a graph database is provided, comprising:

[0012] The relational data corresponding to the edges and nodes is continuously stored based on an append-only mode, and a physical offset value of a scan end position corresponding to the read request is determined based on the read request and the relational data.

[0013] The preset default operator is pre-adjusted according to actual business requirements, and statistical data is collected.

[0014] The target operator is determined from the pre-adjusted default operator according to the statistical data, and the cache function of the target operator is started or stopped, wherein the target operator is an operator that supports caching intermediate results.

[0015] Each node of the target operator is independently saved, maintained versions, and discarded to generate overall cache information.

[0016] An optimal version cache of the target node in the target operator is determined from the overall cache information based on the relational data and the read request, a preset preliminary execution plan is adjusted based on the optimal version cache and the physical offset value, and a corresponding query action is performed.

[0017] According to an aspect of the present application, an electronic device is provided, comprising: a processor; a memory storing a computer program, when the computer program is executed by the processor, the processor executes the method as described above.

[0018] According to an aspect of the present application, a non-transitory computer readable medium is provided, which stores readable instructions, when the instructions are executed by a processor, the processor executes the method as described above.

[0019] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application.

[0020] Beneficial effects:

[0021] Through the above-mentioned embodiments provided by the present application, the statistical analysis module dynamically adjusts the operator cache strategy, enables cache only for high-value operators, and avoids invalid cache from occupying resources. In combination with read requests, the query execution can be quickly located to the optimal version cache, repeated calculation is reduced, and complex queries are significantly accelerated. Based on append-only storage, the historical version of the relational data is retained, the intermediate result cache is bound with the intermediate result version and the read request, and in combination with the physical offset value corresponding to the optimal version cache, only incremental update is required, rather than full reconstruction. The query execution module determines the optimal version cache through the relational data and the read request, and then adjusts the preliminary execution plan, thereby avoiding the cache invalidation problem caused by data changes. In the frequent update of graph data scenarios, the traditional cache scheme has a sharp performance drop due to frequent data changes. The present application uses the append-only storage, the intermediate result cache and the multi-version cache to maintain high query throughput while ensuring data consistency. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings without departing from the scope of the present application.

[0023] Figure 1 A block diagram of a multi-version data cache device based on a graph database provided by an embodiment of the present application;

[0024] Figure 2 A block diagram of another multi-version data cache device based on a graph database provided by an embodiment of the present application;

[0025] Figure 3 A flowchart of a multi-version data cache method based on a graph database provided by an embodiment of the present application;

[0026] Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the drawings, and thus a repeated description will be omitted.

[0028] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0029] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0030] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0031] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below could be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0032] For specific implementation methods, please refer to the following embodiments.

[0033] Figure 1 This is a block diagram of a cache device for multi-version data based on a graph database provided in an embodiment of the present application. Figure 1 As shown, the caching device 10 for multi-version data based on a graph database includes a data storage module 11, a statistical analysis module 12, a cache operator selection module 13, a cache module 14 and a query execution module 15.

[0034] The data storage module 11 is used to continuously store the relationship data corresponding to the edges and nodes based on an append-only method, and determine the physical offset value of the scan end position corresponding to the read request according to the read request and the relationship data.

[0035] In this device, the data storage module 11 can be a graph database storage engine that stores edges based on an append-only mechanism. Graph databases store edges as first-class citizens, and edges to the same node are generally stored contiguously, either physically or logically via pointers. The stored content can be considered relational data. Append-only is a data storage or processing strategy whose core principle is that all new data can only be appended to the end of existing data, and previous data cannot be modified or deleted.

[0036] In a database system, a read request accesses a specific version of the database data. Timestamps, or transaction identifiers (also known as transaction IDs), precisely control the version of the data being read. Timestamps play a key role in concurrency control and multi-version data management, ensuring transaction isolation and consistent reads. To support incremental updates to the cache, similar edges related to the same node are stored in a compact, append-only format. For example, in a social network graph library, one type of edge can be represented by "person_likes_post" associated with a Person node, connecting nodes representing posts that the person has liked; another type of edge can be represented by "person_workAt_organization" connecting nodes representing work organizations. Person_likes_post edges for the same person are stored contiguously, and subsequent updates are simply appended to the edge, including deletions, which are represented by appending a deletion message. Similarly, each person's "person_workAt_organization" edge is stored separately and contiguously.

[0037] When reading a certain type of edge of a node, each read request, in addition to returning normal data, can also return the physical offset value corresponding to the end position of the data scan, that is, the end position of the edge data visible in the latest transaction when the read is initiated.

[0038] The statistical analysis module 12 is used to pre-adjust the preset default operators according to actual business needs and collect statistical data.

[0039] The graph database in this application has many operators, such as breadth-first search, subgraph or path matching, attribute aggregation, n-degree neighbor query, shortest path operator, time range aggregation operator, group counting operator, etc., which can be used as preset default operators. The statistical analysis module 12 can pre-modify the default operator to support reading and writing intermediate result caches according to the granularity of the traversed nodes. This process can be pre-adjusted. In some implementations, the operator can be used to represent a fixed type of computational sub-process in the graph query process.

[0040] In some implementations, since the implementation and calling methods of different default operators of the graph database are greatly affected by the actual business usage and access methods, the usage logic of different default operators of this device for intermediate result caching can be completed in the graph database program development stage, or the interface can be reserved for dynamic loading and modification through plug-ins later.

[0041] The statistical data may include the calling frequency, data input and output scale, and execution cost of different default operators, and these statistical data may be collected by the statistical analysis module 12 .

[0042] The cache operator selection module 13 is used to determine a target operator from the pre-adjusted default operators according to statistical data, and to enable or disable the cache function of the target operator, wherein the target operator is an operator that supports caching intermediate results.

[0043] The cache operator selection module 13 can specifically select an intermediate result cache that can be cached to optimize the cache. In this application, the conditions for operators that support caching intermediate results can be pre-set. By comparing the conditions with statistical data, it is possible to determine which default operators are target operators. Because different read requests correspond to different query processes and involve different target operators, the cache function of the target operator can be turned on or off.

[0044] In some implementations, four constants l, m, n, and t can be pre-configured for each target operator to represent the weight, and a, b, and c are used to represent the operator call frequency in the statistical data, the ratio of the query plan's estimated scan item size to the intermediate result item size, and the median of the histogram of each operator's estimated execution cost output by the optimizer. When la+mb+nc>t, the operator's caching function is enabled.

[0045] In other implementations, the target operator can be randomly selected multiple times using Monte Carlo or all possible choices can be enumerated, and the best option can be enabled based on the execution cost-benefit effect after caching is enabled. The specific process is as follows: First, the user can enter a query that is believed to be frequently accessed in the future, or allow the query intermediate result cache selection and optimization module to read several queries with the highest request frequency on its own. Next, confirm the operators that support caching in the query, and the caching function of each operator is turned on and off independently. For complex queries with a large number of operators that support caching, Monte Carlo can be used to randomly select and enable them multiple times to reduce the number of enumerated combinations. Finally, all combinations are compared from the perspective of cost-benefit before and after the query is turned on and off, the best solution is selected, and the operators in the selected solution are set to have the caching function enabled.

[0046] In other implementations, after the cache function of the target operator is enabled, the cache size and cache hit rate of the target operator can be continuously counted. Three configurable constants p, q, and r can be pre-set, and a, b, and c can be used to represent the call frequency, cache hit rate, and total number of cache entries of the target operator in the statistics respectively. <r时,标记该目标算子可以优先关闭,即后续如果总缓存规模达到预设上线,优先关闭此目标算子的缓存功能,并清理其已有缓存。

[0047] The cache module 14 is used to independently save, version maintain and eliminate each node of the target operator, and has generated overall cache information.

[0048] In this application, the operations within a target operator involve multiple nodes in the graph, and each node can cache multiple versions of the intermediate results of the node-related calculations. New read requests may trigger version updates, so version maintenance is required in different situations. As the amount of stored version data increases, some versions may no longer be used, so it may be necessary to eliminate the versions. In some implementations, the elimination of old versions is not direct deletion, but may be merging inappropriate versions with other versions.

[0049] The cache module 14 can cache multiple versions of intermediate results for each node in the target operator, respond to read requests and update requests, and provide cache eviction functionality. In some implementations, maintenance cycles and eviction conditions can be pre-set, and the cache module 14 performs maintenance and eviction.

[0050] In some implementations, MVCC (Multi-Version Concurrency Control) can be used to support incremental cache updates and multi-version maintenance, ensuring full read and write concurrency. The version read by a read request is based on the timestamp obtained when the request is processed. Data updated by write transactions after this timestamp is not visible to the read request.

[0051] For example, three read requests obtain transaction timestamps t1, t2, and t4 in sequence, and there is an update transaction t3 before t4. The final order of transaction execution to the operator that supports the cache function is t1, t3, t4, and t2. When t1 is executed, a cache version R1 is created. After that, t3 updates the cached node-related edge data and commits it. When t4 is executed, the new data is read and the cache version R2 is updated. When t2 is executed, it can only read the R1 version because t2 should not see the data updated by t3. When implementing MVCC in this application, timestamps can be represented by simple consecutive numbers. Each time a write transaction adds one to this timestamp value, the read transaction timestamp remains unchanged. Each piece of data on the storage side will be marked with the transaction timestamp when it was written. Read operations can only see data that is marked no greater than its own timestamp. In the above example, the read transaction timestamp t1=t2, t3 is the increment of the write transaction timestamp, so t3=t2+1, and then t4 is the read transaction timestamp that remains unchanged, so t4=t3. Therefore, in order to ensure that the query transaction reads the cache correctly, the cache version can be marked with the last modification timestamp of the corresponding data. The timestamp value of the cache version R1 mentioned above is t1 (= t2), and the timestamp of R2 is t4 (= t3).

[0052] There are two different approaches to maintaining a multi-version cache of intermediate results for each node item of a single target operator. One approach, based on the latest version T4, maintains the latest version and a series of rollback logs (undo). Since most read requests read the latest version, the first approach is typically used to maintain the latest version. Each cache update directly calculates the latest cache and records the modification as an undo log. When an older version needs to be read, the undo log is used to roll back the version. That is, if a request wants to obtain cached data for version T4, it can directly call the latest version. If it wants to obtain version T1, it must work backwards from version T4 to obtain version T1.

[0053] The other method is based on the initial version T1, that is, maintaining a relatively old version and a series of incremental data (todo). If you want to get the T1 version, you can directly call the basic initial version.

[0054] For example, in a social network scenario, when searching for the M most active forums among a person A's friends within N degrees, the minimum granularity of the cache is the number of posts by a single friend in each forum. In the above examples of t1, t2, t4, and t3, suppose that after executing t1, the number of posts by the friend in the football, badminton, and basketball forums is 6, 7, and 8 respectively. After that, in transaction t3, the user adds two posts to the football and basketball forums, and posts one post to the bowling forum. Then, after t4, the cache baseline data in the entry corresponding to the friend in the HashMap is {Basketball 10, Football 8, Badminton 7, Bowling 1}. The differential data corresponds to the modification of the intermediate result of the node cache caused by scanning the edges related to the node during the incremental calculation of the operator and calculating on this basis, that is:

[0055] {undo: basketball target scanned and sent after splicing -2, soccer target scanned and sent after splicing -2, bowling target scanned and sent after splicing -1; delete bowling forum item}. When a read request created at time t2 accesses the cache, a replay operation is performed based on the baseline data and the above undo operation, obtaining the old version {basketball 8, badminton 7, soccer 6}. If the undo operation is cumbersome or expensive to execute in the cache of individual operator nodes, the second method can be used to maintain a relatively old cache version. During updates, incremental information is retained to generate the todo message for the target to splice and send after scanning the object, delaying direct modification of the baseline version. In the above example, if the second method is used, the updated cache record after target scanning and splicing is {8 after target scanning and splicing, 7 after badminton target scanning, 6 after soccer target scanning}, and the difference item after target scanning and splicing is {todo: Add bowling forum item; basketball target scanning and splicing +2, soccer target scanning and splicing +2, bowling target scanning and splicing +1}. In this example, the first method is superior. The second method requires periodic merging of the baseline version with some older redo after target scanning and splicing to ensure that the baseline cache does not remain at the oldest version. The first method simply deletes the undo of outdated versions of target scanning and splicing.

[0056] The query execution module 15 is used to determine the optimal version cache corresponding to the target node in the target operator from the overall cache information based on the relational data and the read request, adjust the preset preliminary execution plan based on the optimal version cache and the physical offset value, and execute the corresponding query action.

[0057] In the present application, there are multiple nodes on the graph involved in the calculation process of the target operator, and the read request may include a specific read task. The number and range of nodes corresponding to different tasks are different. The node of the target operator corresponding to the read request can be used as the target node. A preliminary execution plan can be generated in advance, that is, the process of querying according to the original query logic without considering the node cache. The query execution module 15 can select the cache information of the target operator that meets the conditions from the overall cache information based on the relational data corresponding to the read request and the current transaction ID carried by the read request, and can select the optimal version cache based on the preset optimal selection conditions. It should be noted that the optimal in this application refers to the most suitable version that meets the conditions. This version may be the latest version or other versions.

[0058] According to the preset execution plan adjustment method, cached execution logic is added to the original plan. This includes processing the optimal version cache and physical offset value, and then executing the query action corresponding to the adjusted plan. In some methods, an offset vector can be calculated based on the physical offset value, and the optimal version cache and offset vector are processed before executing the query action corresponding to the adjusted plan.

[0059] Traditional caching solutions have difficulty predicting the reuse value of intermediate results of graph queries, resulting in a low cache hit rate. The device of the present application dynamically adjusts the operator caching strategy through a statistical analysis module, and only enables caching for high-value operators to avoid invalid caches occupying resources. Combined with read requests, it ensures that the optimal version cache can be quickly located when the query is executed, reducing repeated calculations and significantly accelerating complex queries. Based on append-only storage, the historical version of the relational data is retained, and the intermediate result cache is bound to the intermediate result version and the read request. In conjunction with the physical offset value corresponding to the optimal version cache, only incremental updates are required, rather than full reconstruction. The query execution module determines the optimal version cache through relational data and read requests, and then adjusts the preliminary execution plan to avoid cache invalidation problems caused by data changes. In the scenario of frequently updated graph data, the performance of traditional caching solutions drops sharply due to frequent data changes. The present application maintains high query throughput while ensuring data consistency through append-only storage, intermediate result caching, and multi-version caching.

[0060] According to some embodiments, reference Figure 2 The caching device 10 for multi-version data based on a graph database also includes a materialization loading module 16, which is used to materialize the overall cache information of the target operator according to a preset materialization interval, so as to reload the materialized overall cache information when the service is restarted.

[0061] The materialized loading module 16 can be a module that materializes and reloads cached intermediate results. To prevent cache loss due to process exit or hardware restart, the cache can be materialized periodically. This can be done on this module or on other modules or devices. After the service restarts, the materialized cache can be loaded and reused. The materialization interval can be pre-set by the user based on actual needs.

[0062] The device in this application periodically materializes the cache information of the target operator, persisting intermediate results in memory (e.g., to disk or distributed storage), ensuring rapid loading after a service restart and avoiding performance degradation caused by cold starts. In a production environment requiring high availability, services may be restarted due to failures or upgrades. By materializing cache information, the device in this application allows the system to directly reuse historical calculation results after recovery, reducing manual intervention and operational pressure.

[0063] According to some embodiments, when the scanning operation corresponding to the read request is scanning multiple types of edges, the data storage module 11 can obtain the physical offset values ​​corresponding to each of the multiple types of edges to generate an offset vector based on the physical offset values, where the multiple types of edges are edges corresponding to the same target operator.

[0064] In this application, since the target operator corresponds to multiple nodes and each node in the graph database is connected to multiple types of edges, the read request may scan multiple types of edges at the same time. For example, when obtaining the number of posts and the number of posts a person likes, it is necessary to scan the edges that record the posts he likes and the edges that record his posting information. Therefore, multiple offsets need to be obtained to form an offset vector.

[0065] The data storage module 11 can maintain an append-only log categorized by edge type, with each edge write accompanied by a globally incrementing logical timestamp (or version number). For each edge type (e.g., edge types E1, E2, ...En), the module records its current maximum physical offset value (i.e., the disk file offset or log sequence number of the last write). In response to a read request, the incremental scan range is obtained. The physical offset ranges of multiple edge types can be encoded as structured data as physical offset values, which can then be input into a pre-defined structure, outputting the corresponding offset vector.

[0066] The device of the present application accurately records the data update position of each type of edge through the physical offset value, so that the query only scans the newly added or changed data segments, avoiding redundant I / O operations and significantly improving scanning efficiency.

[0067] According to some embodiments, the cache module 14 caches intermediate results according to a preset granularity of nodes within the operator.

[0068] In this application, graph queries usually start from a node and traverse the nodes through the node's adjacent edges. When traversing the nodes, the operators in the query often read a large amount of information around the nodes in the graph involved for calculation, and the calculation results of these nodes are likely to be reused by other queries later. Therefore, intermediate results can be cached at the node granularity to improve the cache reuse rate. HashMap (Hash-Map) or other easy-to-retrieve data structures are usually used to organize and manage the specific intermediate result content corresponding to each node in the same operator.

[0069] For example, in a social networking scenario, a common query is to search for the M most active forums among person A's N-degree friends. If only the entire query result is cached, the cache is only valid for the specific parameters A, N, and M, which is very inefficient. During the first query traversal, this application caches the posting information for all of A's N-degree friends separately and stores it in a HashMap. Subsequent queries for the posting information of person B's K-degree friends can query the HashMap for each friend and attempt to directly retrieve the cached information. If a friend of B's ​​is not in the cache, their posting information is obtained through conventional methods and then added to the cache. At this point, each HashMap entry represents the posting information for each friend. Each entry can use a BTreeMap (B-Tree Map) or red-black tree to store the number of posts by a friend on different forums. This cache can also be used for other types of queries, such as querying for the top three-degree friends of person C who have the highest overlap with C's top ten favorite forums, as the posting information of C's three-degree friends is likely already cached.

[0070] The device of the present application caches according to node-level granularity (such as single node, neighbor set, subgraph), so that different queries can reuse the intermediate results of the same node, significantly improving the hit rate.

[0071] According to some embodiments, when the cache operator selection module 13 determines the target operator from the pre-adjusted default operators based on statistical data, it is specifically used to: determine the cache hit rate and cache activation benefit of the pre-adjusted default operator based on statistical data; and determine the default operator whose cache hit rate and cache activation benefit meet the preset cache activation conditions as the target operator.

[0072] In this application, the cache hit rate refers to the percentage of points that can be directly read from the cache, based on the cache information read around the points involved in the calculation process of the target operator with the cache function enabled. For example, if an operator retrieves the latest comment posts for a certain type of person, it will use the ID of each person that meets the conditions to query the corresponding cache. The percentage of successful cache reads compared to the total number of queries is the cache hit rate.

[0073] In some implementations, the number of queries and the number of successful queries can be accumulated to obtain the actual cumulative hit rate. In other implementations, the hit rate can be simply estimated using the cache size and data size. For example, if the Person data size is 10,000 and the cache size is 1,000, the hit rate can be estimated as 10%.

[0074] The benefit of turning on the cache means that after the cache function is turned on, the target operator can skip part of the original execution because it can use the cache, thereby saving the query execution cost. The query execution plan generation process uses a dynamic programming algorithm to construct a tree plan, and the entire generation optimization process is based on a cost model. The model estimates the resource consumption of different execution paths (such as I / O operations, CPU time, etc.) and selects the execution plan with the lowest estimated cost. In this application, in the optimization using the cache, for any target operator, once the cache function is turned on, in its subsequent calculations around the node, the part that hits the cache can skip execution and directly read the intermediate results of the corresponding point from the cache, thereby saving resource consumption (such as I / O operations, CPU time, etc.). For all the resources saved by this operator minus the cache maintenance cost, the benefit of turning on the cache is the benefit of turning on the cache.

[0075] Two parts of this application utilize the calculation of the benefits of enabling the cache. One part is used to evaluate the effectiveness of enabling the cache of the target operator. In order to facilitate the evaluation of whether a target operator should enable or disable the cache function, the cumulative hit benefits of each time the cache function of the target operator is used can be counted. The cumulative value is affected by many factors, including the number of calls, the cache hit rate in each call, and the cost that can be saved by actually skipping steps once a hit occurs in each call. For operators with low cumulative hit benefits, the query intermediate result cache selection and optimization module will give them priority to the shutdown and elimination operations. The other part is used to determine whether to enable the cache function of the target operator, that is, to use Monte Carlo to randomly select multiple times for high-frequency queries or to enumerate all possible choices, and to enable the cache function of the target operator based on the execution cost-benefit effect after enabling the cache.

[0076] When calculating the benefits of enabling caching, you can use historical hit rates or estimate cache hit rates based on data size, while also considering the additional time cost of updating the cache. The benefits of enabling a single operator within a query are calculated as follows:

[0077] The original execution cost without caching enabled - the execution cost after skipping some steps after caching enabled * cache hit rate - the cost of updating and maintaining the cache.

[0078] During the specific implementation process, the above parameters can be obtained in advance.

[0079] If caching is enabled for multiple operators within a query, the benefit of caching is not the sum of the individual benefits of the operators. This is because if several operators are in the same pipeline, caching on a lower-level operator reduces the original execution cost of an upper-level operator without caching or with a cache miss. Therefore, the benefit calculated using the above method needs to be corrected. This correction is done by subtracting the benefit of the downstream caching operator from the benefit of the upstream operator.

[0080] Based on real-time statistics of cache hit rate and revenue data, the device in this application only enables caching for target operators that meet the conditions, avoiding invalid cache occupying resources. By continuously monitoring the cache hit rate and the revenue from enabling cache, the cache strategy is automatically adjusted to adapt to changes in query patterns.

[0081] According to some embodiments, when the cache module 14 independently stores the target operator and its internal nodes, it is specifically used to store the target operator based on the preset calculation type of the target operator.

[0082] In this application, caches for the same type of computation can be stored uniformly, with each target operator stored independently. During the storage process, the target operator can be stored as a node ID. Therefore, each target operator can be stored separately according to its computation type. Furthermore, target operators can be stored in a detailed manner, with versions corresponding to each node being divided based on each node's ID.

[0083] In some implementations, an append-only mechanism can be used to independently store node versions in the target operator. In some implementations, a globally monotonically increasing transaction timestamp can be used to ensure system-wide uniqueness. When a target operator (such as a path search operator) generates an intermediate result, the current timestamp T_i (i.e., the timestamp of the current read request) is bound to the generated function key result. This sets a unique transaction timestamp for the corresponding version, allowing for the storage of multiple versions of the intermediate result.

[0084] The present invention supports independent storage of each node of a target operator based on the computation type, independently maintaining version information, avoiding resource waste caused by unified storage and improving memory / cache utilization. Furthermore, centralized storage of operators of the same computation type facilitates rapid retrieval and reuse of computed results through the principle of cache locality, reducing repeated computations. This is particularly suitable for loop structures or frequently called operators.

[0085] According to some embodiments, when the cache module 14 performs elimination processing on each node of the target operator, it is specifically used to: obtain the number of versions corresponding to each node; when the number of versions is greater than the preset cache limit, obtain the transaction timestamps corresponding to multiple versions of each node; and merge the versions whose transaction timestamps meet the preset merging conditions.

[0086] In this application, the cache module 14 stores multiple target operators. Each node corresponding to each target operator has multiple versions. The number of versions of the node can be directly called. The cache limit can be pre-set based on historical business volume and the usage of different versions. Merge conditions can also be pre-set, which can be used to characterize the method of data merging.

[0087] For each node, the number of versions corresponding to that node is compared with the cache limit. If the number of versions is less than or equal to the cache limit, no action is required. If the number of versions is greater than the cache limit, the transaction timestamps of different versions can be checked and the versions whose transaction timestamps meet the merge conditions can be merged.

[0088] In some implementations, there are versions 0, 1, 2, and 3, with version 3 being the version closest to the current time. A merge condition may be that if version 0 is to be eliminated, version 0 may be merged with version 1 or with both version 1 and version 2 first.

[0089] In other implementations, the timestamps of various versions may be directly detected, elimination conditions may be pre-set, and versions whose timestamps satisfy the elimination conditions may be merged with the latest version or an adjacent version.

[0090] In other implementations, it can be added that each time the operator updates the cache, it will check the transaction ID (transaction timestamp) of the smallest version in the currently active transaction, and then check whether the oldest batch of cached versions in the cache module lags behind the minimum version transaction ID above. When there are multiple lagging versions, they can be merged into one lagging version. For example, if the current active transaction is t30, and it is found that t9, t19, t29, and t39 exist in the cache, then the t9 and t19 versions can be eliminated. The reason for retaining the t29 version here is to confirm that if the t30 transaction accesses the cache, it can obtain the appropriate version based on the t29 version.

[0091] The cache module can passively (such as triggered by insufficient preset cache memory) or actively eliminate operator caches with low cache value and outdated cache versions (among the n caches that are older than the time version corresponding to the oldest active transaction, the oldest n-1 will not be accessed, and this part of the cache should be eliminated. If different version caches use incremental / differential storage, the information in the eliminated versions should be merged with the remaining versions; if the device strictly limits the number of cache versions, when the cache version reaches the limit, it will also trigger the cleanup of the old version. If there are still old transactions accessing the cache afterwards, and all existing cache versions are newer than the timestamp corresponding to the access, the cache cannot be used).

[0092] The device of the present application can perform overall elimination processing on the target operator cache to avoid target operators that have not been used for a long time from occupying resources and improve query efficiency.

[0093] According to some embodiments, the query execution module 15 is specifically used to: when the optimal version cache is not empty and the current timestamp corresponding to the read request is less than the transaction timestamp of the oldest version of the target node, determine that the cache read is empty and execute the query according to the preliminary execution plan; when the optimal version cache is not empty and the current timestamp is equal to the transaction timestamp of the version of the target node, execute the query based on the optimal version cache; when the optimal version cache is not empty and the current timestamp is greater than the transaction timestamp of the latest version of the target node, or the current timestamp corresponding to the read request is between the transaction timestamps corresponding to different versions of the target node, select the cache information of the optimal version of the node with a transaction timestamp less than the current timestamp from the optimal version cache; obtain the physical offset value of the version with a timestamp less than the current timestamp, and perform a supplementary query based on the physical offset value to determine the cache information difference, and execute the query based on the cache information difference and the cache information; when the optimal version cache is empty, execute the query according to the preliminary execution plan.

[0094] In this application, since the overall cache data corresponding to the target operator includes each node in the operator and each version corresponding to each node, the optimal version cache can be extracted from multiple versions of the target node first, and targeted analysis can be performed on the operator node cache and corresponding query actions can be executed.

[0095] First, when reading the intermediate result cache of a specified node of a specified target operator, the read request must be accompanied by the current timestamp T_j of the current read query, which is returned by the cache module 14. The cache module 14 internally maintains multiple versions of the corresponding intermediate results, and each version is bound to the transaction timestamp when it was generated. That is, the transaction timestamp of the corresponding version is set based on the current timestamp T_j. In response to the read request, the query execution module 15 has four types of processing methods:

[0096] The first type: If the optimal version cache is not empty and the current timestamp corresponding to the read request is less than the transaction timestamp of the oldest version of the target operator, the cache read is empty and the query needs to be executed according to the preliminary execution plan.

[0097] The second method: If the optimal version cache is not empty and the current timestamp is equal to the transaction timestamp of the target node's version, the cache information of that version is directly returned.

[0098] The third method: If the optimal version cache is not empty and the current timestamp is greater than the transaction timestamp of the latest version of the target operator, or the current timestamp corresponding to the read request is between the transaction timestamps corresponding to different versions of the target operator, then the cache of the latest version T_i with a timestamp less than T_j is returned, along with the offset value D_i corresponding to version T_i in the append-only file system. Subsequent queries read all visible data (i.e., the cache information delta) starting from the append-only file D_i, combining the cached and newly acquired data to complete the query. In some implementations, a supplementary query is performed based on the physical offset vector to determine the cache information delta.

[0099] The fourth method: When the new cache information is empty, the query is executed directly according to the preliminary execution plan.

[0100] This application performs targeted queries based on the latest cache information, current timestamp, and timestamps of multiple versions of the target operator, thereby improving query efficiency.

[0101] According to some embodiments, when the cache module 14 independently saves each node of the target operator, it is specifically used to: when the optimal version cache is not empty and the current timestamp is greater than the transaction timestamp of the latest version of the target node, store the new intermediate result version in the query process of executing the query according to the preliminary execution plan to update multiple versions of the target node; when the optimal version cache is not empty and the current timestamp corresponding to the read request is between the transaction timestamps corresponding to different versions of the target node, or the current timestamp is greater than or equal to the timestamp of the latest version of the target node, determine that multiple versions of the target node do not need to be updated; when the optimal version cache is empty, store the intermediate result version in the query process of executing the query according to the preliminary execution plan.

[0102] In the present application, the query execution module 15 has different query modes, and the cache module 14 has corresponding different data update modes.

[0103] If the optimal version cache is not empty and the current timestamp is greater than the transaction timestamp of the latest version, the query process can be executed directly according to the preliminary execution plan, and the newly generated intermediate result version can be stored to update multiple versions of the corresponding target node.

[0104] If the optimal version cache is not empty, and the current timestamp corresponding to the read request is between the transaction timestamps corresponding to different versions of the target node, or the current timestamp is greater than or equal to the transaction timestamp of the latest version of the target node, there is no need to update the version.

[0105] If the latest cache information is empty, the query execution module 15 executes the query according to the preliminary execution plan, and the cache module 14 can directly store the intermediate result version generated during the query process.

[0106] This application performs targeted version updates of target nodes under different query modes of the query execution module, and only recalculates incremental data, thereby reducing the usage of computing resources.

[0107] Figure 3 This is a flowchart of a method for caching multi-version data based on a graph database provided in an embodiment of the present application. The method of this embodiment can be applied to the above-mentioned caching device for multi-version data based on a graph database. Figure 3 As shown, the method includes: step S30, step S31, step S32, step S33 and step S34.

[0108] In step S30 , the relationship data corresponding to the edges and nodes are continuously stored based on the append-only method, and the physical offset value of the scan end position corresponding to the read request is determined according to the read request and the relationship data.

[0109] In step S31, the preset default operator is pre-adjusted according to actual business needs, and statistical data is collected.

[0110] In step S32, a target operator is determined from the pre-adjusted default operators based on statistical data, and a cache function of the target operator is turned on or off, wherein the target operator is an operator that supports caching of intermediate results.

[0111] In step S33, each node of the target operator is independently saved, versioned, and eliminated to generate overall cache information.

[0112] In step S34, based on the relational data and the read request, the optimal version cache corresponding to the target node in the target operator is determined from the overall cache information, so as to adjust the preset preliminary execution plan based on the optimal version cache and the physical offset value, and execute the corresponding query action.

[0113] In this application, the data storage module can pre-store edge and node relationship data and physical offset values. Users can send read requests through the user request module. The statistical analysis module can obtain relevant statistical data and pre-adjust the current default operator. The adjusted default operator and statistical data can then be sent to the cache operator selection module. The cache operator selection module can select target operators that support caching intermediate results based on statistical data and control the opening or closing of these target operators.

[0114] The query execution module attempts to obtain the latest cached information from the cache module and execute the query to achieve fast query speed. If no results are found, the query is performed according to the preliminary execution plan without using cached intermediate results. The query results are sent to the user and the target operator and corresponding intermediate result version used in the query are returned to the cache operator selection module for update. This in turn updates the cached version / the first intermediate result of the target operator in the cache module.

[0115] The materialized loading module can periodically materialize the intermediate results cached in the cache module and reload them at startup.

[0116] The method performs functions similar to those of the device provided above. For other functions, please refer to the above description and will not be repeated here.

[0117] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device 400 of this embodiment may include: a memory 401 and a processor 402.

[0118] The memory 401 stores a computer program. When the computer program is executed by the processor 402 , the processor 402 executes the method in the above embodiment.

[0119] The processor 402 and the memory 401 are connected, for example, via a bus.

[0120] Optionally, the electronic device 400 may further include a transceiver. It should be noted that in actual applications, the number of transceivers is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0121] Processor 402 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 402 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0122] A bus includes a path that transmits information between the components mentioned above. Examples include a PCI (Peripheral Component Interconnect) bus and an EISA (Extended Industry Standard Architecture) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the diagram uses a single thick line, but this does not imply a single bus or type of bus.

[0123] The memory 401 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0124] The memory 401 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 402. The processor 402 is used to execute the application code stored in the memory 401 to implement the content shown in the above method embodiment.

[0125] Electronic devices include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. They may also include servers, etc. Figure 4 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0126] The electronic device of this embodiment can be used to execute the method of any of the above embodiments. Its implementation principles and technical effects are similar and will not be described in detail here.

[0127] The present application also provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon. When the aforementioned instructions are executed by a processor, the processor executes the method in the above embodiment.

[0128] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0129] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. At the same time, changes or modifications made by those skilled in the art based on the ideas of the present application, the specific implementation methods, and the scope of application of the present application, all fall within the scope of protection of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. A multi-version data caching device based on a graph database, characterized in that: include: a data storage module configured to continuously store relationship data corresponding to edges and nodes in an append-only manner, and determine a physical offset value of a scan end position corresponding to a read request based on a read request and the relationship data; A statistical analysis module is used to pre-adjust the preset default operators according to actual business needs and collect statistical data; wherein the statistical data includes the call frequency, data input and output scale, and execution cost of different default operators; a cache operator selection module, configured to determine a target operator from the pre-adjusted default operators based on the statistical data, and to enable or disable a cache function of the target operator, wherein the target operator is an operator that supports caching intermediate results; A cache module, configured to independently save, version maintain, and eliminate each node of the target operator to generate overall cache information; a query execution module, configured to determine, based on the relational data and the read request, an optimal version cache corresponding to a target node in the target operator from the overall cache information, adjust a preset preliminary execution plan based on the optimal version cache and a physical offset value, and execute a corresponding query action; Wherein, when the cache operator selection module determines the target operator from the pre-adjusted default operators according to the statistical data, it is further configured to: Determine the cache hit rate and cache enablement benefit of the pre-adjusted default operator based on the statistical data; A default operator whose cache hit rate and cache activation benefit satisfy a preset cache activation condition is determined as the target operator.

2. The device according to claim 1, characterized in that It also includes a materialization loading module, which is used to materialize the entire cache information according to a preset materialization interval, so as to reload the materialized entire cache information when the service is restarted.

3. The device according to claim 1, characterized in that In the case where the scanning operation corresponding to the read request is scanning multiple types of edges, the data storage module obtains the physical offset values ​​corresponding to each of the multiple types of edges to generate an offset vector based on the physical offset values, wherein the multiple types of edges are edges corresponding to the same target operator.

4. The device according to claim 1, characterized in that The cache module caches the intermediate results according to a preset granularity of the node.

5. The device according to claim 1, characterized in that When the cache module independently stores each node of the target operator, it is specifically used to: The target operator is stored based on a preset calculation type of the target operator.

6. The device according to claim 5, characterized in that When the cache module performs elimination processing on each operator of the target operator, it is specifically used to: Obtain the version number corresponding to each operator; When the number of versions is greater than the preset cache limit, obtaining the transaction timestamps corresponding to the multiple versions of each operator; The versions whose transaction timestamps meet the preset merging conditions are merged.

7. The device according to claim 1, characterized in that The query execution module is specifically used to: If the optimal version cache is not empty and the current timestamp corresponding to the read request is less than the transaction timestamp of the oldest version of the target node, determine that the cache read is empty, and execute the query according to the preliminary execution plan; If the optimal version cache is not empty and the current timestamp is equal to the transaction timestamp of the version of the target node, execute the query based on the optimal version cache; If the optimal version cache is not empty and the current timestamp is greater than the transaction timestamp of the latest version of the target node, or the current timestamp corresponding to the read request is between the transaction timestamps corresponding to different versions of the target node, select the cache information of the optimal version of the node whose transaction timestamp is less than the current timestamp from the optimal version cache; Obtaining the physical offset value of the version with a timestamp less than the current timestamp, performing a supplementary query based on the physical offset value to determine a cache information difference, and performing a query based on the cache information difference and the cache information; When the optimal version cache is empty, the query is executed according to the preliminary execution plan.

8. The device according to claim 7, characterized in that When performing version maintenance on each node of the target operator, the cache module is specifically used to: If the optimal version cache is not empty and the current timestamp is greater than the transaction timestamp of the latest version of the target node, storing a new intermediate result version in the query process of executing the query according to the preliminary execution plan to update the multiple versions of the target node; If the optimal version cache is not empty and the current timestamp corresponding to the read request is between the transaction timestamps corresponding to different versions of the target node, or the current timestamp is greater than or equal to the transaction timestamp of the latest version of the target node, determining that the multiple versions of the target node do not need to be updated; When the optimal version cache is empty, the intermediate result versions during the query process of executing the query according to the preliminary execution plan are stored.

9. A multi-version data caching method based on a graph database, characterized in that: include: Continuously storing relationship data corresponding to edges and nodes in an append-only manner, and determining a physical offset value of a scan end position corresponding to a read request based on the read request and the relationship data; Pre-adjust the preset default operators based on actual business needs and collect statistical data; the statistical data includes the call frequency, data input and output scale, and execution cost of different default operators; Determine a target operator from the pre-adjusted default operators according to the statistical data, and enable or disable a cache function of the target operator, wherein the target operator is an operator that supports caching of intermediate results; Performing independent storage, version maintenance, and elimination processing on each node of the target operator to generate overall cache information; Based on the relationship data and the read request, determine the optimal version cache corresponding to the target node in the target operator from the overall cache information, adjust a preset preliminary execution plan based on the optimal version cache and the physical offset value, and execute the corresponding query action; The step of determining a target operator from the pre-adjusted default operators based on the statistical data includes: Determine the cache hit rate and cache enablement benefit of the pre-adjusted default operator based on the statistical data; A default operator whose cache hit rate and cache activation benefit satisfy a preset cache activation condition is determined as the target operator.

Images