Multi-layer nested query cache reuse method and system based on dynamic materialization strategy

Abstract

The present invention provides a multi-layer nested query cache reuse method and system based on dynamic materialization strategy, which relates to the field of cache reuse technology, including constructing a query dependency graph, dynamically materializing and marking based on the ratio of query cost to materialization cost; generating a result fingerprint including query characteristics, version information and timestamp for the materialized node, and storing the query result in a distributed cache system of a hierarchical architecture; when processing a new query, efficient query result reuse is achieved by verifying the version information and dependency data of the cache result. The present invention can significantly improve query performance, reduce computing resource consumption, and achieve efficient caching and reuse of query results.

Description

Technical Field

[0001] The present invention relates to cache reuse technology, and in particular to a multi-layer nested query cache reuse method and system based on a dynamic materialization strategy. Background Art

[0002] With the rapid development of big data and cloud computing technologies, query requirements in enterprise data analysis applications are becoming increasingly complex, and multi-layer nested queries have become a common query mode. In this query mode, a complex query usually contains multiple interdependent subqueries, and the upper query needs to use the results of the lower query as input. In order to improve query performance and response speed, the cache and reuse mechanism of query results has become particularly important.

[0003] At present, database systems generally use materialized view technology to cache and reuse query results. Traditional materialized views are pre-defined and maintained in the database, and the system automatically updates the content of the materialized view according to preset rules. However, this static materialization method has some technical problems and shortcomings when processing dynamic multi-layer nested queries.

[0004] The existing technology mainly has the following defects and shortcomings: when processing multi-layer nested queries, there is a lack of systematic analysis and management of the dependencies between query nodes, and it is impossible to effectively determine which intermediate results are suitable for materialized caching, resulting in inability to optimally utilize system resources.

[0005] Existing cache systems usually adopt a unified storage strategy and fail to perform hierarchical storage management based on the access characteristics of query results, which leads to mixed storage of high-frequency access data and low-frequency access data, affecting the overall performance and resource utilization efficiency of the cache system.

[0006] In terms of the reuse of cached results, the existing technology lacks a complete version control and data consistency verification mechanism, and is unable to accurately track and verify the validity of cached results, especially when dealing with multi-layer nested queries with complex dependencies, which is prone to data inconsistency problems. Summary of the invention

[0007] The embodiments of the present invention provide a multi-layer nested query cache reuse method and system based on a dynamic materialization strategy, which can solve the problems in the prior art.

[0008] A first aspect of an embodiment of the present invention provides a multi-layer nested query cache reuse method based on a dynamic materialization strategy, comprising:

[0009] A query dependency graph is constructed according to the received multi-layer nested query request, wherein the query dependency graph includes multiple query nodes and data dependency relationships between the multiple query nodes, and each query node corresponds to a subquery; based on the query dependency graph, a query cost and a materialized cost are calculated for each query node, and a ratio of the query cost to the materialized cost is defined as a materialized benefit ratio, and multiple query nodes are dynamically materialized according to the materialized benefit ratio;

[0010] For each query node marked as materialized, the corresponding query result is obtained and a result fingerprint is generated. The result fingerprint includes the query statement characteristics of the query node, data source version information, and query timestamp; the query result and result fingerprint are stored in the distributed cache system, and a version association relationship is established between the query node and the upstream query node it depends on; the distributed cache system adopts a layered storage architecture, storing frequently accessed query results in the memory layer and infrequently accessed query results in the disk layer;

[0011] When a new query request is received, the query statement features in the new query request are extracted, and the cached results with the same query statement features are searched in the distributed cache system; if the cached result is found, it is verified whether the data source version information of the cached result is consistent with the current data source version, and based on the version association relationship, it is determined whether the dependent data of the cached result has changed; when the cached result verification is valid, the cached result is returned as a response to the new query request, and the access frequency information of the cached result is updated at the same time; when the cached result verification is invalid, the query is re-executed and the cache is updated.

[0012] Based on the query dependency graph, the query cost and materialization cost are calculated for each query node. The ratio of the query cost to the materialization cost is defined as the materialization benefit ratio. Dynamic materialization marking of multiple query nodes according to the materialization benefit ratio includes:

[0013] Receive a multi-layer nested query request, build a query dependency graph based on the multi-layer nested query request, the query dependency graph includes a query node set and a dependency edge set, when the output result of a first query node is used as the input of a second query node, establish a directed edge between the first query node and the second query node, the directed edge belongs to the dependency edge set; identify strongly connected components in the query dependency graph by a depth-first search algorithm, and eliminate circular dependencies;

[0014] For each query node in the query node set, the query cost is calculated based on the weighted sum of data access cost, intermediate result calculation cost and result transmission cost, wherein the data access cost includes input and output cost and processor cost, the intermediate result calculation cost includes data processing overhead and operation overhead, and the result transmission cost includes network transmission delay;

[0015] The materialization cost is calculated by weighted summing up the storage space overhead, the maintenance overhead, and the failure detection overhead, wherein the storage space overhead is determined by the result set size, the maintenance overhead is determined by the data update cost, and the failure detection overhead is determined by the version verification cost; the materialization benefit ratio is determined by multiplying the ratio of the query cost to the materialization cost and the query frequency factor, wherein the query frequency factor is calculated based on the number of cache hits, the total number of queries, and the time decay coefficient;

[0016] Based on the current materialized memory usage, the maximum available memory and the expected memory usage, the materialization threshold is adaptively adjusted using the learning rate; when the materialization benefit ratio is greater than the materialization threshold, the corresponding query node is marked as a materialized node.

[0017] For each query node marked as materialized, obtain the corresponding query result and generate a result fingerprint. The result fingerprint includes the query statement characteristics of the query node, data source version information, and query timestamp, including:

[0018] Receive a query statement, and perform normalization processing on the query statement, wherein the normalization processing includes removing spaces and line break characters in the query statement, unifying the uppercase and lowercase formats in the query statement, and converting constant values ​​in the query statement into parameter forms; construct a query feature vector based on the normalized query statement, wherein the query feature vector includes a plurality of feature weights, and each feature weight is obtained by multiplying a feature frequency by an inverse document frequency of a feature and performing normalization processing;

[0019] Obtaining query-related data source version information, the data source version information including a basic version number and an incremental change identifier, wherein the basic version number is obtained by string concatenating the version numbers of the tables involved and calculating a hash value, and the incremental change identifier is obtained by weighting the weight coefficients of each of the tables involved and the most recent modification timestamp; generating a composite timestamp including the result creation time, the expected expiration time, and the most recent access time, wherein the expected expiration time is calculated based on the data update cycle and the query mode change cycle;

[0020] The query feature vector, the data source version information and the composite timestamp are combined and a secure hash value is calculated to obtain a result fingerprint, the result fingerprint is compressed using an adaptive multi-level local sensitive hashing method, the number of hash buckets is dynamically adjusted in combination with a dynamic weight matrix, and hot and cold tiered storage is performed according to the heat value of the result fingerprint.

[0021] The result fingerprint is compressed by an adaptive multi-level local sensitive hashing method, the number of hash buckets is dynamically adjusted in combination with a dynamic weight matrix, and hot and cold hierarchical storage is performed according to the heat value of the result fingerprint, including:

[0022] Divide the query statements into first-layer query feature vector groups according to similar query pattern features, divide them into second-layer version information groups according to associated version information, and divide them into third-layer timestamp groups according to adjacent time windows; count the cache hit rates of the first-layer query feature vector groups, the second-layer version information groups, and the third-layer timestamp groups, and calculate a dynamic weight matrix based on the gradient of the cache hit rate, wherein the dynamic weight matrix includes query feature weights, version information weights, and timestamp weights;

[0023] Based on the dynamic weight matrix, weighted fusion is performed on the first-layer query feature vector grouping, the second-layer version information grouping, and the third-layer timestamp grouping to generate a comprehensive feature representation; the number of hash collisions and the total number of queries based on the comprehensive feature representation are counted, and the number of hash buckets is dynamically adjusted according to the ratio of the number of hash collisions to the total number of queries;

[0024] Calculate the query pattern similarity, version similarity and time correlation, and weight the query pattern similarity, version similarity and time correlation based on a preset weight coefficient to obtain a comprehensive similarity score; when a feature change is detected, combine the original result fingerprint with the feature of the changed part and calculate the hash value to obtain an updated result fingerprint;

[0025] The access time and access frequency of each result fingerprint are counted, and the heat value of each result fingerprint is calculated based on the time decay factor; the result fingerprint is hierarchically stored according to the heat value, and the hierarchical storage includes a hot data layer and a cold data layer; when the heat value of the result fingerprint in the hot data layer is lower than a first preset threshold, the result fingerprint is migrated to the cold data layer; when the heat value of the result fingerprint in the cold data layer is higher than a second preset threshold, the result fingerprint is migrated to the hot data layer; when the heat value of the result fingerprint is lower than a third preset threshold, the result fingerprint is cleared from the storage.

[0026] Verify whether the data source version information of the cached result is consistent with the current data source version, and determine whether the dependent data of the cached result has changed based on the version association relationship, including:

[0027] Construct a data source version vector and a version timestamp matrix, wherein the data source version vector includes the version numbers of multiple data sources, and the version timestamp matrix includes the last update time of each attribute in each data source; calculate the weighted difference between the cached version number and the current version number based on the preset data source weight to obtain the version difference value, and obtain the maximum difference between the current time and the cached time in the version timestamp matrix to obtain the time difference value;

[0028] Construct a dependency matrix to represent the dependency relationship between data sources, and calculate the dependency propagation coefficient between data sources based on the dependency matrix, where the dependency propagation coefficient is obtained by continuous multiplication of the dependency degree on the dependency path; for a target data source, calculate the direct impact score of the target data source affected by the data source version change based on the dependency matrix, and calculate the propagation impact score of the target data source affected by the data source version change based on the dependency propagation coefficient;

[0029] Substitute the version difference value and the time difference value into an exponential decay function to obtain a consistency score, and substitute the weighted sum of the direct impact score and the propagation impact score into an exponential decay function to obtain a dependency stability score; multiply the consistency score by the dependency stability score and compare them with a preset validity threshold to determine whether the cache result is valid;

[0030] An update necessity index is calculated based on the consistency score and the dependency stability score, and the data source whose comprehensive influence exceeds the update threshold is determined as the update range; the cache failure rate is counted, and the preset validity threshold is updated based on the ratio of the cache failure rate to the target failure rate; the historical hit rate of each data source is counted, and the preset data source weight is updated based on the difference between the historical hit rate and the average hit rate.

[0031] Calculating the direct impact score of the target data source being affected by the data source version change based on the dependency matrix, and calculating the propagation impact score of the target data source being affected by the data source version change based on the dependency propagation coefficient includes:

[0032] Obtain the new version number and the old version number of the target data source, calculate the difference between the new version number and the old version number, and normalize the difference based on the old version number to obtain the version change amount; obtain the access frequency of the target data source, calculate the data source weight based on the access frequency, and the data source weight maps the access frequency to a preset interval through logarithmic operation; multiply the version change amount with the data source weight and the data source dependency strength, and calculate the direct impact score in combination with the data source importance score;

[0033] Construct a propagation path based on the dependency relationship between data sources, calculate the dependency strength product of each edge on the propagation path, and introduce a length decay coefficient to obtain the path weight; multiply the path weight with the weighted sum of the version changes of all target data sources on the path and the data source weight to obtain the path importance; identify the key propagation path based on the mean and standard deviation of the path importance;

[0034] The propagation level coefficient is multiplied by the path importance of all paths in the corresponding level and the results are accumulated to obtain a propagation impact score; a time decay function value is calculated based on the time difference, wherein the time decay function includes a periodic adjustment term; the time decay function value is weighted with the data source association strength to obtain a timing impact strength; a basic score is obtained by weighted combination of the direct impact score and the propagation impact score; and the basic score is adjusted based on the timing impact strength to obtain a final propagation impact score.

[0035] A second aspect of an embodiment of the present invention provides a multi-layer nested query cache reuse system based on a dynamic materialization strategy, including:

[0036] The first unit is used to construct a query dependency graph according to the received multi-layer nested query request, wherein the query dependency graph includes multiple query nodes and data dependency relationships between the multiple query nodes, and each query node corresponds to a subquery; based on the query dependency graph, the query cost and the materialized cost are calculated for each query node, the ratio of the query cost to the materialized cost is defined as the materialized benefit ratio, and the multiple query nodes are dynamically materialized according to the materialized benefit ratio;

[0037] The second unit is used to obtain the corresponding query results and generate result fingerprints for each query node marked as materialized. The result fingerprints include the query statement characteristics of the query node, data source version information, and query timestamp; store the query results and result fingerprints in the distributed cache system, and establish a version association relationship between the query node and the upstream query node it depends on; the distributed cache system adopts a layered storage architecture, storing the query results with high frequency access in the memory layer and the query results with low frequency access in the disk layer;

[0038] The third unit is used to extract the query statement features in the new query request when a new query request is received, and search for the cached result with the same query statement features in the distributed cache system; if the cached result is found, verify whether the data source version information of the cached result is consistent with the current data source version, and determine whether the dependent data of the cached result has changed based on the version association relationship; when the cached result is verified to be valid, return the cached result as a response to the new query request, and update the access frequency information of the cached result; when the cached result is verified to be invalid, re-execute the query and update the cache.

[0039] A third aspect of the embodiments of the present invention

[0040] An electronic device is provided, comprising:

[0041] processor;

[0042] a memory for storing processor-executable instructions;

[0043] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0044] A fourth aspect of the embodiments of the present invention is:

[0045] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0046] The beneficial effects of this application are as follows:

[0047] The present invention makes dynamic materialization decisions by calculating the materialization benefit ratio based on the query dependency graph, fully considering the balance between query cost and materialization cost, avoiding unnecessary materialization overhead, and improving the overall query performance and resource utilization efficiency of the system.

[0048] The present invention adopts the result fingerprint mechanism and version association relationship to track the validity of cache results, ensuring the consistency and timeliness of cache data. At the same time, through the hierarchical storage architecture of the distributed cache system, the query results are stored in the memory layer and the disk layer respectively according to the access frequency, realizing the reasonable allocation of cache storage resources.

[0049] When processing new query requests, the present invention can quickly locate and verify reusable cache results and timely update invalid caches through query statement feature matching and version verification mechanisms, which significantly reduces the execution overhead of repeated queries, improves query response speed, and enhances the query processing capabilities of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a multi-layer nested query cache reuse method based on a dynamic materialization strategy according to an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the structure of a multi-layer nested query cache reuse system based on a dynamic materialization strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0054] Figure 1 FIG. 1 is a flow chart of a multi-layer nested query cache reuse method based on a dynamic materialization strategy according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0055] S101. Construct a query dependency graph according to the received multi-layer nested query request, the query dependency graph includes multiple query nodes and data dependency relationships between the multiple query nodes, each query node corresponds to a subquery; based on the query dependency graph, calculate the query cost and materialized cost for each query node, define the ratio of the query cost to the materialized cost as the materialized benefit ratio, and dynamically materialize the multiple query nodes according to the materialized benefit ratio;

[0056] S102. For each query node marked as materialized, obtain the corresponding query result and generate a result fingerprint, which includes the query statement characteristics of the query node, data source version information and query timestamp; store the query result and the result fingerprint in the distributed cache system, and establish a version association relationship between the query node and the upstream query node it depends on; the distributed cache system adopts a layered storage architecture, storing the query results with high frequency access in the memory layer and the query results with low frequency access in the disk layer;

[0057] S103. When a new query request is received, the query statement features in the new query request are extracted, and the cached results with the same query statement features are searched in the distributed cache system; if the cached result is found, it is verified whether the data source version information of the cached result is consistent with the current data source version, and based on the version association relationship, it is determined whether the dependent data of the cached result has changed; when the cached result is verified to be valid, the cached result is returned as a response to the new query request, and the access frequency information of the cached result is updated at the same time; when the cached result is verified to be invalid, the query is re-executed and the cache is updated.

[0058] In an optional implementation, based on the query dependency graph, the query cost and the materialized cost are calculated for each query node, the ratio of the query cost to the materialized cost is defined as the materialized benefit ratio, and the dynamic materialization marking of multiple query nodes according to the materialized benefit ratio includes:

[0059] Receive a multi-layer nested query request, build a query dependency graph based on the multi-layer nested query request, the query dependency graph includes a query node set and a dependency edge set, when the output result of a first query node is used as the input of a second query node, establish a directed edge between the first query node and the second query node, the directed edge belongs to the dependency edge set; identify strongly connected components in the query dependency graph by a depth-first search algorithm, and eliminate circular dependencies;

[0060] For each query node in the query node set, the query cost is calculated based on the weighted sum of data access cost, intermediate result calculation cost and result transmission cost, wherein the data access cost includes input and output cost and processor cost, the intermediate result calculation cost includes data processing overhead and operation overhead, and the result transmission cost includes network transmission delay;

[0061] The materialization cost is calculated by weighted summing up the storage space overhead, the maintenance overhead, and the failure detection overhead, wherein the storage space overhead is determined by the result set size, the maintenance overhead is determined by the data update cost, and the failure detection overhead is determined by the version verification cost; the materialization benefit ratio is determined by multiplying the ratio of the query cost to the materialization cost and the query frequency factor, wherein the query frequency factor is calculated based on the number of cache hits, the total number of queries, and the time decay coefficient;

[0062] Based on the current materialized memory usage, the maximum available memory and the expected memory usage, the materialization threshold is adaptively adjusted using the learning rate; when the materialization benefit ratio is greater than the materialization threshold, the corresponding query node is marked as a materialized node.

[0063] In a distributed data processing system, we first receive multi-layer nested query requests submitted by users. For complex nested queries, we build a query dependency graph to represent the relationship between queries. For example, when the query "find the top ten product categories in terms of sales" contains a subquery "to calculate the total sales of each product category", two query nodes are created in the dependency graph, and a directed edge is established from the subquery to the outer query. We use depth-first search to identify circular dependencies, such as when query A depends on the result of query B, query B depends on query C, and query C depends on query A. We need to break this cycle.

[0064] When calculating the query cost for each query node, multiple factors are considered. The data access cost includes the input and output time of reading the data file and the time for the processor to perform operations such as scanning and filtering. Taking the scanning of the order table of tens of millions as an example, it takes 10 seconds to read the data file sequentially and 5 seconds for the processor to perform the filter condition judgment. The cost of calculating the intermediate results includes the data processing overhead such as sorting, aggregation and other operations, as well as the computational overhead such as numerical calculation and string processing. The result transmission cost is calculated based on the data volume and network bandwidth. For example, it takes 10 seconds to transmit 1GB of data at a bandwidth of 100MB / s.

[0065] When calculating the materialization cost, the storage space overhead is determined by the size of the result set, such as 1GB of space is required to store 1 million rows of order detail data. The maintenance overhead depends on the frequency and cost of data updates, such as once an hour, a single update takes 5 minutes. Failure detection requires regular verification of whether the cached data version is up to date, such as checking once every 10 minutes, a single check takes 1 second. The materialization benefit ratio is adjusted based on the query frequency factor, which is calculated based on historical access data. For example, if a query result has been reused 100 times in the past hour, its materialization priority is increased.

[0066] The system dynamically adjusts the materialization threshold. When the current memory usage is 8GB, the maximum available memory is 16GB, and the expected usage rate is 80%, if the current materialization benefit ratio is greater than the threshold of 2.0, the query result will be materialized into the memory. As the system runs, the threshold is adjusted based on the actual effect with a learning rate of 0.1 to make memory usage more reasonable.

[0067] This application enables:

[0068] By building a query dependency graph and eliminating circular dependencies, the relationship between complex queries is made clearer, infinite loops are avoided, the reliability and stability of query execution are improved, and the risk of system failure is reduced.

[0069] Using a multi-dimensional cost model to calculate query cost and materialization cost, and dynamically adjusting the materialization strategy based on query frequency, can accurately identify the most valuable query results for caching, improve cache hit rate, reduce repeated calculations, and significantly improve query performance.

[0070] By adaptively adjusting the materialized threshold mechanism, efficient use of memory resources is achieved, memory overflow is avoided, query acceleration effects are maximized while ensuring stable system operation, and the overall system throughput and response speed are improved.

[0071] In an optional implementation, for each query node marked as materialized, the corresponding query result is obtained and a result fingerprint is generated, the result fingerprint includes the query statement features of the query node, the data source version information and the query timestamp including:

[0072] Receive a query statement, and perform normalization processing on the query statement, wherein the normalization processing includes removing spaces and line break characters in the query statement, unifying the uppercase and lowercase formats in the query statement, and converting constant values ​​in the query statement into parameter forms; construct a query feature vector based on the normalized query statement, wherein the query feature vector includes a plurality of feature weights, and each feature weight is obtained by multiplying a feature frequency by an inverse document frequency of a feature and performing normalization processing;

[0073] Obtaining query-related data source version information, the data source version information including a basic version number and an incremental change identifier, wherein the basic version number is obtained by string concatenating the version numbers of the tables involved and calculating a hash value, and the incremental change identifier is obtained by weighting the weight coefficients of each of the tables involved and the most recent modification timestamp; generating a composite timestamp including the result creation time, the expected expiration time, and the most recent access time, wherein the expected expiration time is calculated based on the data update cycle and the query mode change cycle;

[0074] The query feature vector, the data source version information and the composite timestamp are combined and a secure hash value is calculated to obtain a result fingerprint, the result fingerprint is compressed using an adaptive multi-level local sensitive hashing method, the number of hash buckets is dynamically adjusted in combination with a dynamic weight matrix, and hot and cold tiered storage is performed according to the heat value of the result fingerprint.

[0075] For the result fingerprint generation process of the materialized query node, the input query statement needs to be standardized and preprocessed first. For example, for the query statement "SELECT user_id, COUNT(*) as cnt FROM user_log WHEREcreate_time>'2024-01-01' GROUP BY user_id", first remove the spaces and line breaks in the statement, convert all keywords to uppercase, and replace the string constant '2024-01-01' with the parameter form '?' to obtain the standardized query statement "SELECTuser_id,COUNT(*)ascntFROMuser_logWHEREcreate_time>?GROUPBYuser_id".

[0076] Next, we construct the query feature vector by analyzing the frequency of each feature in the normalized query statement and combining it with the inverse document frequency of the feature in the historical query. For the example query, the extracted features include select operation, count aggregation, group by grouping, etc. Assuming that the feature "select" appears once in the query and 8,000 times in the 10,000 historical queries, its feature weight is 0.125; the feature "count" appears once and 2,000 times in history, and its feature weight is 0.5. All feature weights are normalized to obtain the final query feature vector.

[0077] When obtaining data source version information, it is necessary to collect the version information of the user_log table involved in the query. Assuming that the current version number of the user_log table is "v2.1.3", perform hash calculation to obtain the basic version number; at the same time, the table was last modified at "2024-02-23 10:00:00", and the weight coefficient is 0.8, based on which the incremental change identifier is calculated. Combined with the current time "2024-02-23 14:00:00" as the result creation time, and based on the fact that the user_log table is updated once a day, the expected expiration time is set to "2024-02-24 10:00:00".

[0078] Finally, the query feature vector, version information, and timestamp information are combined, and the SHA-256 algorithm is used to calculate the complete result fingerprint. Considering the storage efficiency, the local sensitive hashing method is used to compress the result fingerprint. During compression, the number of hash buckets is dynamically adjusted according to the current system load. For example, when the load is low, 1024 hash buckets are used, and when the load is high, it is expanded to 4096 hash buckets. At the same time, the result fingerprint of the hot query is stored in the memory, and the cold data is stored in the disk to achieve hierarchical storage.

[0079] This application enables:

[0080] The first part is reflected in query optimization. Through the normalization of query statements and the construction of feature vectors, it can accurately identify equivalent queries, effectively improve the reuse rate of query results, and reduce repeated calculation overhead. At the same time, the dynamic weight scheme can better adapt to changes in query patterns.

[0081] The second part is about ensuring data consistency. By introducing data source version information and a composite timestamp mechanism, it is able to perceive data changes in a timely manner and respond to them, ensuring the consistency of cached results with source data and avoiding errors caused by using expired data.

[0082] The third part is about the utilization of system resources. The adaptive multi-level local sensitive hashing and hot and cold tiered storage strategies can reduce storage space usage while ensuring query performance and improve the overall resource utilization of the system. Dynamically adjust parameters according to load conditions to further optimize system performance.

[0083] In an optional implementation, the result fingerprint is compressed using an adaptive multi-level local sensitive hashing method, the number of hash buckets is dynamically adjusted in combination with a dynamic weight matrix, and hot and cold hierarchical storage is performed according to the heat value of the result fingerprint, including:

[0084] Divide the query statements into first-layer query feature vector groups according to similar query pattern features, divide them into second-layer version information groups according to associated version information, and divide them into third-layer timestamp groups according to adjacent time windows; count the cache hit rates of the first-layer query feature vector groups, the second-layer version information groups, and the third-layer timestamp groups, and calculate a dynamic weight matrix based on the gradient of the cache hit rate, wherein the dynamic weight matrix includes query feature weights, version information weights, and timestamp weights;

[0085] Based on the dynamic weight matrix, weighted fusion is performed on the first-layer query feature vector grouping, the second-layer version information grouping, and the third-layer timestamp grouping to generate a comprehensive feature representation; the number of hash collisions and the total number of queries based on the comprehensive feature representation are counted, and the number of hash buckets is dynamically adjusted according to the ratio of the number of hash collisions to the total number of queries;

[0086] Calculate the query pattern similarity, version similarity and time correlation, and weight the query pattern similarity, version similarity and time correlation based on a preset weight coefficient to obtain a comprehensive similarity score; when a feature change is detected, combine the original result fingerprint with the feature of the changed part and calculate the hash value to obtain an updated result fingerprint;

[0087] The access time and access frequency of each result fingerprint are counted, and the heat value of each result fingerprint is calculated based on the time decay factor; the result fingerprint is hierarchically stored according to the heat value, and the hierarchical storage includes a hot data layer and a cold data layer; when the heat value of the result fingerprint in the hot data layer is lower than a first preset threshold, the result fingerprint is migrated to the cold data layer; when the heat value of the result fingerprint in the cold data layer is higher than a second preset threshold, the result fingerprint is migrated to the hot data layer; when the heat value of the result fingerprint is lower than a third preset threshold, the result fingerprint is cleared from the storage.

[0088] First, the input query statement is analyzed in multiple dimensions. According to the query pattern characteristics, the syntax tree parsing method is used to extract the structural characteristics of the query statement, such as table connection relationship, query conditions, etc. For example, for the query statement "SELECT product_name FROM orders WHERE order_date>'2024-01-01'", single table query, time range filtering and other features are extracted. Based on similar features, the queries are divided into different groups, such as "order query group" and "user information query group".

[0089] For the version information feature, extract the version number, update time and other information of the data table involved in the query. For example, the version number of the order table is "v2.1" and the last update time is "2024-02-01". Group the query by version information, such as "v2.1 version group", "v2.0 version group", etc.

[0090] For time features, set a sliding time window (such as 1 hour) to count the time distribution of queries. For example, divide the queries from 0:00 to 1:00 into one group, and the queries from 1:00 to 2:00 into another group.

[0091] Dynamically monitor the cache hits of each group. Assume that the hit rate of the query feature group is 85%, the hit rate of the version group is 75%, and the hit rate of the time group is 65%. Calculate the dynamic weights based on the trend of the hit rate changes. The current weights are 0.4, 0.3, and 0.3 respectively.

[0092] Feature fusion and dynamic adjustment of hash buckets;

[0093] The features of each layer are weighted and combined by dynamic weights. For example, the feature vector of a query is [0.8, 0.6, 0.5], and multiplied by the weight [0.4, 0.3, 0.3] to obtain the weighted feature [0.32, 0.18, 0.15].

[0094] Count the hash collision rate, that is, the proportion of hash values ​​that are the same but the actual query is different. When the collision rate exceeds 20%, increase the number of hash buckets, and when it is less than 5%, reduce the number of buckets. For example, if the current number of hash buckets is 1000 and the collision rate is 25%, adjust the number of buckets to 1500.

[0095] Similarity calculation and result fingerprint update;

[0096] Calculate the similarity between queries. Query pattern similarity is calculated based on syntax tree structure comparison, version similarity is calculated based on version number difference, and time similarity is calculated based on time interval. For example, if the syntax tree structure similarity of two queries is 0.9, the version similarity is 0.8, and the time similarity is 0.7, the weighted comprehensive similarity is 0.8.

[0097] Update the result fingerprint when a query feature change is detected. For example, when the WHERE condition changes from "order_date>'2024-01-01'" to "order_date>'2024-02-01'", keep the common features and recalculate the hash value only for the changed part.

[0098] Heat calculation and hierarchical storage management;

[0099] Track and record the access history of each result fingerprint. For example, a result fingerprint has been accessed 100 times in the past hour, and the last access was 10 minutes ago. Set the time decay factor to 0.9, and the current heat value is 90.

[0100] Storage is layered based on heat values. The hot data layer stores data with heat values ​​higher than 80, using memory storage; the cold data layer stores data with heat values ​​between 20-80, using solid-state drives. When the heat value is lower than 20, the data is cleaned up.

[0101] Check data heat changes regularly. When the heat of a result fingerprint in the hot data layer drops to 75, it is transferred to the cold data layer; when the heat of a result fingerprint in the cold data layer rises to 85, it is transferred to the hot data layer.

[0102] This application can improve the accuracy of query pattern recognition through multi-level feature analysis and dynamic weight calculation, so that the result fingerprint can more accurately express the query characteristics, reduce storage space usage, and improve the overall performance of the system.

[0103] Based on the dynamic adjustment mechanism of hash buckets, on-demand allocation of storage resources is achieved, the probability of hash conflicts is reduced, query efficiency is improved, and the scalability and stability of the system are enhanced.

[0104] The use of a heat-aware tiered storage strategy achieves reasonable scheduling of storage resources, improves the response speed of high-frequency queries, reduces storage costs, and optimizes the system's resource utilization efficiency.

[0105] In an optional implementation, verifying whether the data source version information of the cached result is consistent with the current data source version, and determining whether the dependent data of the cached result has changed based on the version association relationship includes:

[0106] Construct a data source version vector and a version timestamp matrix, wherein the data source version vector includes the version numbers of multiple data sources, and the version timestamp matrix includes the last update time of each attribute in each data source; calculate the weighted difference between the cached version number and the current version number based on the preset data source weight to obtain the version difference value, and obtain the maximum difference between the current time and the cached time in the version timestamp matrix to obtain the time difference value;

[0107] Construct a dependency matrix to represent the dependency relationship between data sources, and calculate the dependency propagation coefficient between data sources based on the dependency matrix, where the dependency propagation coefficient is obtained by continuous multiplication of the dependency degree on the dependency path; for a target data source, calculate the direct impact score of the target data source affected by the data source version change based on the dependency matrix, and calculate the propagation impact score of the target data source affected by the data source version change based on the dependency propagation coefficient;

[0108] Substitute the version difference value and the time difference value into an exponential decay function to obtain a consistency score, and substitute the weighted sum of the direct impact score and the propagation impact score into an exponential decay function to obtain a dependency stability score; multiply the consistency score by the dependency stability score and compare them with a preset validity threshold to determine whether the cache result is valid;

[0109] An update necessity index is calculated based on the consistency score and the dependency stability score, and the data source whose comprehensive influence exceeds the update threshold is determined as the update range; the cache failure rate is counted, and the preset validity threshold is updated based on the ratio of the cache failure rate to the target failure rate; the historical hit rate of each data source is counted, and the preset data source weight is updated based on the difference between the historical hit rate and the average hit rate.

[0110] First, build a data source version management system. For multiple data sources in the system, record the version information of each data source, including the version number and attribute-level timestamp information. For data source A, its version number can be an increasing integer, such as the current version is 238, and the last update time of each field is "user name: 2024-02-20 10:30:00", "age: 2024-02-21 15:20:00", etc. Organize the version information of all data sources into version vectors, such as the version vectors of five data sources are [238, 156, 89, 442, 267]. At the same time, establish a version timestamp matrix to record the last update time of each attribute in each data source.

[0111] Next, evaluate the cache version consistency. Based on the pre-configured data source weights (such as [0.3, 0.2, 0.15, 0.2, 0.15]), calculate the weighted difference between the cache version number and the current version number. For example, if the cache version vector is [236, 156, 89, 440, 267], the version difference value can be obtained. At the same time, extract the maximum difference between the current time and the cache time from the version timestamp matrix as the time difference value, such as the maximum time difference is 72 hours.

[0112] Construct a data source dependency network. The dependency matrix represents the dependency strength between data sources, with a value range of 0 to 1. For example, if the dependency from data source A to B is 0.8, it means that 80% of the data in A depends on B. The propagation coefficient is calculated based on the dependency matrix, which is obtained by continuously multiplying the dependencies on the dependency path. For example, if A depends on B (0.8) and B depends on C (0.6), the propagation coefficient of A affected by C is 0.48. For the target data source, the direct impact score is calculated in combination with the version change, and the propagation impact score is calculated based on the propagation coefficient.

[0113] Evaluate cache effectiveness. Substitute the version difference value and the time difference value into the exponential decay function to get the consistency score. For example, if the version difference value is 5 and the time difference value is 72 hours, the consistency score may be 0.85. Similarly, substitute the weighted sum of the direct impact score and the propagation impact score into the exponential decay function to get a dependency stability score, such as 0.92. Multiply the two scores and compare them with the effectiveness threshold (such as 0.75) to determine whether the cache is effective.

[0114] Finally, adaptive optimization is performed. The necessity of updating is calculated based on the consistency score and the dependency stability score, and the data sources with an impact exceeding the threshold (such as 0.3) are identified as the update range. The cache failure rate is calculated. For example, if the recent failure rate is 15% and the target failure rate is 10%, the validity threshold is adjusted accordingly. At the same time, the historical hit rate of the data source is calculated. For example, if the hit rate of a data source is 85%, which is higher than the average hit rate of 75%, the weight of the data source is appropriately increased.

[0115] This application can achieve:

[0116] Through the dual verification mechanism of version vector and timestamp matrix, combined with the data source weight, accurate version consistency assessment can be achieved, which not only ensures the timeliness of cached results, but also avoids excessive updates, thereby improving system performance and resource utilization efficiency.

[0117] The analysis method based on dependency matrix and propagation coefficient comprehensively evaluates the direct and indirect impact of data source version changes, realizes accurate modeling of complex dependency relationships, improves the accuracy of cache update decisions, and reduces system response delays.

[0118] Adaptive threshold and weight adjustment strategies are adopted to dynamically optimize cache management parameters, so that the system can automatically adjust strategies according to actual operating conditions, maximize cache effects while ensuring data consistency, and enhance system availability and stability.

[0119] In an optional implementation, calculating the direct impact score of the target data source being affected by the data source version change based on the dependency matrix, and calculating the propagation impact score of the target data source being affected by the data source version change based on the dependency propagation coefficient includes:

[0120] Obtain the new version number and the old version number of the target data source, calculate the difference between the new version number and the old version number, and normalize the difference based on the old version number to obtain the version change amount; obtain the access frequency of the target data source, calculate the data source weight based on the access frequency, and the data source weight maps the access frequency to a preset interval through logarithmic operation; multiply the version change amount with the data source weight and the data source dependency strength, and calculate the direct impact score in combination with the data source importance score;

[0121] Construct a propagation path based on the dependency relationship between data sources, calculate the dependency strength product of each edge on the propagation path, and introduce a length decay coefficient to obtain the path weight; multiply the path weight with the weighted sum of the version changes of all target data sources on the path and the data source weight to obtain the path importance; identify the key propagation path based on the mean and standard deviation of the path importance;

[0122] The propagation level coefficient is multiplied by the path importance of all paths in the corresponding level and the results are accumulated to obtain a propagation impact score; a time decay function value is calculated based on the time difference, wherein the time decay function includes a periodic adjustment term; the time decay function value is weighted with the data source association strength to obtain a timing impact strength; a basic score is obtained by weighted combination of the direct impact score and the propagation impact score; and the basic score is adjusted based on the timing impact strength to obtain a final propagation impact score.

[0123] First, analyze and evaluate the version changes of the target data source. Obtain the old and new version numbers of the target data source through the version management system. For example, the old version is 2.1.0 and the new version is 2.4.0. Calculate the difference of the version numbers and normalize them. Divide the version difference 3 by the old version number 2.1 to get the version change of 1.43. At the same time, obtain the recent access frequency data of the data source from the system log, assuming that the average number of visits is 1,000 per day. Map the access frequency to the range of 0-1 through logarithmic operation to obtain the data source weight of 0.8.

[0124] Then, the dependency strength between data sources is analyzed. The dependency matrix shows that the target data source has a dependency strength of 0.7 with the upstream data source. Combined with the importance score of 0.9 for the data source, the version change of 1.43, the data source weight of 0.8, the dependency strength of 0.7, and the importance of 0.9, the direct impact score of 0.72 is obtained.

[0125] Then, we construct the propagation paths between data sources. Starting from the target data source, we extend upstream along the dependency relationship to form multiple propagation paths. For each path, we calculate the dependency strength product of the edges and introduce a length attenuation coefficient of 0.8. For example, the dependency strengths of a three-layer propagation path are 0.7, 0.6, and 0.5, respectively, and the path weight is 0.7×0.6×0.5×0.8=0.168. We weight the path weight with the version changes and weights of each data source on the path to obtain a path importance of 0.25.

[0126] The importance of all propagation paths is counted, and the mean is 0.2 and the standard deviation is 0.05. Paths that exceed one standard deviation from the mean are identified as key propagation paths. Attenuation coefficients are set for different propagation levels, with the first level being 1, the second level being 0.8, and the third level being 0.6. The level coefficient is multiplied by the importance of all paths in that level and then added up to obtain a propagation impact score of 0.58.

[0127] The time decay value is calculated based on the time difference of the data source version change. A periodic adjustment item is introduced, and a period is set to 7 days. When the current time difference is 3 days, the decay value is 0.8. The time decay value is multiplied by the data source association strength 0.75 to obtain the time series impact strength 0.6. Finally, the direct impact score 0.72 and the communication impact score 0.58 are combined with a weight of 6:4 to obtain a basic score of 0.66, and then the basic score is adjusted in combination with the time series impact strength 0.6, and finally the communication impact score is 0.4.

[0128] This application enables:

[0129] Comprehensiveness of data impact assessment: By analyzing multiple dimensions such as version changes, access frequency, dependencies, etc. of the data source, combined with a comprehensive assessment of direct impact and propagation impact, it is possible to comprehensively measure the impact of data source version changes and avoid missing important influencing factors.

[0130] Accuracy of evaluation results: By introducing adjustment factors such as data source weight, dependency strength, and time decay, and through multi-level propagation path analysis, the evaluation results are made more in line with the actual situation, thereby improving the accuracy and reliability of impact assessment.

[0131] Practicality of system operation and maintenance: Based on the evaluation results, potential risks brought by data source version changes can be discovered in a timely manner, providing decision-making basis for system operation and maintenance and version management, helping to establish an early warning mechanism for data source version changes, and improving system stability.

[0132] Figure 2 FIG. 1 is a schematic diagram of the structure of a multi-layer nested query cache reuse system based on a dynamic materialization strategy according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0133] The first unit is used to construct a query dependency graph according to the received multi-layer nested query request, wherein the query dependency graph includes multiple query nodes and data dependency relationships between the multiple query nodes, and each query node corresponds to a subquery; based on the query dependency graph, the query cost and the materialized cost are calculated for each query node, the ratio of the query cost to the materialized cost is defined as the materialized benefit ratio, and the multiple query nodes are dynamically materialized according to the materialized benefit ratio;

[0134] The second unit is used to obtain the corresponding query results and generate result fingerprints for each query node marked as materialized. The result fingerprints include the query statement characteristics of the query node, data source version information, and query timestamp; store the query results and result fingerprints in the distributed cache system, and establish a version association relationship between the query node and the upstream query node it depends on; the distributed cache system adopts a layered storage architecture, storing the query results with high frequency access in the memory layer and the query results with low frequency access in the disk layer;

[0135] The third unit is used to extract the query statement features in the new query request when a new query request is received, and search for the cached result with the same query statement features in the distributed cache system; if the cached result is found, verify whether the data source version information of the cached result is consistent with the current data source version, and determine whether the dependent data of the cached result has changed based on the version association relationship; when the cached result is verified to be valid, return the cached result as a response to the new query request, and update the access frequency information of the cached result; when the cached result is verified to be invalid, re-execute the query and update the cache.

[0136] According to a third aspect of the embodiments of the present invention,

[0137] An electronic device is provided, comprising:

[0138] processor;

[0139] a memory for storing processor-executable instructions;

[0140] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0141] A fourth aspect of the embodiments of the present invention is:

[0142] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0143] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-layer nested query cache reuse method based on dynamic materialization strategy, characterized in that: include: Building a query dependency graph according to the received multi-layer nested query request, the query dependency graph includes multiple query nodes and data dependency relationships between the multiple query nodes, each query node corresponds to a subquery; Based on the query dependency graph, the query cost and materialization cost are calculated for each query node. The ratio of query cost to materialization cost is defined as the materialization benefit ratio. Multiple query nodes are dynamically materialized according to the materialization benefit ratio. For each query node marked as materialized, obtain the corresponding query result and generate a result fingerprint, which includes the query statement characteristics of the query node, data source version information, and query timestamp; The query results and result fingerprints are stored in the distributed cache system, and a version association relationship is established between the query node and the upstream query node it depends on. The distributed cache system adopts a layered storage architecture, storing frequently accessed query results in the memory layer and infrequently accessed query results in the disk layer. When a new query request is received, the query statement features in the new query request are extracted, and the cache results with the same query statement features are searched in the distributed cache system; If a cached result is found, verify whether the data source version information of the cached result is consistent with the current data source version, and determine whether the dependent data of the cached result has changed based on the version association relationship; When the cached result is verified to be valid, the cached result is returned as a response to a new query request, and the access frequency information of the cached result is updated at the same time; when the cached result is verified to be invalid, the query is re-executed and the cache is updated; Verify whether the data source version information of the cached result is consistent with the current data source version, and determine whether the dependent data of the cached result has changed based on the version association relationship, including: Construct a data source version vector and a version timestamp matrix, wherein the data source version vector includes the version numbers of multiple data sources, and the version timestamp matrix includes the last update time of each attribute in each data source; calculate the weighted difference between the cached version number and the current version number based on the preset data source weight to obtain the version difference value, and obtain the maximum difference between the current time and the cached time in the version timestamp matrix to obtain the time difference value; Construct a dependency matrix to represent the dependency relationship between data sources, and calculate the dependency propagation coefficient between data sources based on the dependency matrix, where the dependency propagation coefficient is obtained by continuous multiplication of the dependency degree on the dependency path; for a target data source, calculate the direct impact score of the target data source affected by the data source version change based on the dependency matrix, and calculate the propagation impact score of the target data source affected by the data source version change based on the dependency propagation coefficient; Substitute the version difference value and the time difference value into an exponential decay function to obtain a consistency score, and substitute the weighted sum of the direct impact score and the propagation impact score into an exponential decay function to obtain a dependency stability score; multiply the consistency score by the dependency stability score and compare them with a preset validity threshold to determine whether the cache result is valid; An update necessity index is calculated based on the consistency score and the dependency stability score, and data sources exceeding the update threshold are used as the update range; cache failure rate is counted, and the preset validity threshold is updated based on the ratio of the cache failure rate to the target failure rate; the historical hit rate of each data source is counted, and the preset data source weight is updated based on the difference between the historical hit rate and the average hit rate.

2. The method according to claim 1, characterized in that: Based on the query dependency graph, the query cost and materialization cost are calculated for each query node. The ratio of the query cost to the materialization cost is defined as the materialization benefit ratio. Dynamic materialization marking of multiple query nodes according to the materialization benefit ratio includes: Receive a multi-layer nested query request, build a query dependency graph based on the multi-layer nested query request, the query dependency graph includes a query node set and a dependency edge set, when the output result of a first query node is used as the input of a second query node, establish a directed edge between the first query node and the second query node, the directed edge belongs to the dependency edge set; identify strongly connected components in the query dependency graph by a depth-first search algorithm, and eliminate circular dependencies; For each query node in the query node set, the query cost is calculated based on the weighted sum of data access cost, intermediate result calculation cost and result transmission cost, wherein the data access cost includes input and output cost and processor cost, the intermediate result calculation cost includes data processing overhead and operation overhead, and the result transmission cost includes network transmission delay; The materialization cost is calculated by weighted summing up the storage space overhead, the maintenance overhead, and the failure detection overhead, wherein the storage space overhead is determined by the result set size, the maintenance overhead is determined by the data update cost, and the failure detection overhead is determined by the version verification cost; the materialization benefit ratio is determined by multiplying the ratio of the query cost to the materialization cost and the query frequency factor, wherein the query frequency factor is calculated based on the number of cache hits, the total number of queries, and the time decay coefficient; Based on the current materialized memory usage, the maximum available memory and the expected memory usage, the materialization threshold is adaptively adjusted using the learning rate; when the materialization benefit ratio is greater than the materialization threshold, the corresponding query node is marked as a materialized node.

3. The method according to claim 1, characterized in that For each query node marked as materialized, obtain the corresponding query result and generate a result fingerprint. The result fingerprint includes the query statement characteristics of the query node, data source version information, and query timestamp, including: Receive a query statement, and perform normalization processing on the query statement, wherein the normalization processing includes removing spaces and line break characters in the query statement, unifying the uppercase and lowercase formats in the query statement, and converting constant values ​​in the query statement into parameter forms; construct a query feature vector based on the normalized query statement, wherein the query feature vector includes a plurality of feature weights, and each feature weight is obtained by multiplying a feature frequency by an inverse document frequency of a feature and performing normalization processing; Obtaining query-related data source version information, the data source version information including a basic version number and an incremental change identifier, wherein the basic version number is obtained by string concatenating the version numbers of the tables involved and calculating a hash value, and the incremental change identifier is obtained by weighting the weight coefficients of each of the tables involved and the most recent modification timestamp; generating a composite timestamp including the result creation time, the expected expiration time, and the most recent access time, wherein the expected expiration time is calculated based on the data update cycle and the query mode change cycle; The query feature vector, the data source version information and the composite timestamp are combined and a secure hash value is calculated to obtain a result fingerprint, the result fingerprint is compressed using an adaptive multi-level local sensitive hashing method, the number of hash buckets is dynamically adjusted in combination with a dynamic weight matrix, and hot and cold tiered storage is performed according to the heat value of the result fingerprint.

4. The method according to claim 3, characterized in that The result fingerprint is compressed by an adaptive multi-level local sensitive hashing method, the number of hash buckets is dynamically adjusted in combination with a dynamic weight matrix, and hot and cold hierarchical storage is performed according to the heat value of the result fingerprint, including: Divide the query statements into first-layer query feature vector groups according to similar query pattern features, divide them into second-layer version information groups according to associated version information, and divide them into third-layer timestamp groups according to adjacent time windows; count the cache hit rates of the first-layer query feature vector groups, the second-layer version information groups, and the third-layer timestamp groups, and calculate a dynamic weight matrix based on the gradient of the cache hit rate, wherein the dynamic weight matrix includes query feature weights, version information weights, and timestamp weights; Based on the dynamic weight matrix, weighted fusion is performed on the first-layer query feature vector grouping, the second-layer version information grouping, and the third-layer timestamp grouping to generate a comprehensive feature representation; the number of hash collisions and the total number of queries based on the comprehensive feature representation are counted, and the number of hash buckets is dynamically adjusted according to the ratio of the number of hash collisions to the total number of queries; Calculate the query pattern similarity, version similarity and time correlation, and weight the query pattern similarity, version similarity and time correlation based on a preset weight coefficient to obtain a comprehensive similarity score; when a feature change is detected, combine the original result fingerprint with the feature of the changed part and calculate the hash value to obtain an updated result fingerprint; The access time and access frequency of each result fingerprint are counted, and the heat value of each result fingerprint is calculated based on the time decay factor; the result fingerprint is hierarchically stored according to the heat value, and the hierarchical storage includes a hot data layer and a cold data layer; when the heat value of the result fingerprint in the hot data layer is lower than a first preset threshold, the result fingerprint is migrated to the cold data layer; when the heat value of the result fingerprint in the cold data layer is higher than a second preset threshold, the result fingerprint is migrated to the hot data layer; when the heat value of the result fingerprint is lower than a third preset threshold, the result fingerprint is cleared from the storage.

5. The method according to claim 1, characterized in that Calculating the direct impact score of the target data source being affected by the data source version change based on the dependency matrix, and calculating the propagation impact score of the target data source being affected by the data source version change based on the dependency propagation coefficient includes: Obtain the new version number and the old version number of the target data source, calculate the difference between the new version number and the old version number, and normalize the difference based on the old version number to obtain the version change amount; obtain the access frequency of the target data source, calculate the data source weight based on the access frequency, and the data source weight maps the access frequency to a preset interval through logarithmic operation; multiply the version change amount with the data source weight and the data source dependency strength, and calculate the direct impact score in combination with the data source importance score; Construct a propagation path based on the dependency relationship between data sources, calculate the dependency strength product of each edge on the propagation path, and introduce a length decay coefficient to obtain the path weight; multiply the path weight with the weighted sum of the version changes of all target data sources on the path and the data source weight to obtain the path importance; identify the key propagation path based on the mean and standard deviation of the path importance; The propagation level coefficient is multiplied by the path importance of all paths in the corresponding level and the results are accumulated to obtain the propagation impact score; the time decay function value is calculated based on the time difference, wherein the time decay function includes a periodic adjustment term, and the time decay function value is weighted with the data source association strength to obtain the timing impact strength; the direct impact score and the propagation impact score are weightedly combined to obtain a basic score, and the basic score is adjusted based on the timing impact strength to obtain the final propagation impact score.

6. A multi-layer nested query cache reuse system based on dynamic materialization strategy, used to implement the method as described in any one of claims 1 to 5, characterized in that: include: The first unit is used to construct a query dependency graph according to the received multi-layer nested query request, the query dependency graph includes multiple query nodes and data dependency relationships between the multiple query nodes, and each query node corresponds to a subquery; Based on the query dependency graph, the query cost and materialization cost are calculated for each query node. The ratio of query cost to materialization cost is defined as the materialization benefit ratio. Multiple query nodes are dynamically materialized according to the materialization benefit ratio. The second unit is used to obtain the corresponding query result and generate a result fingerprint for each query node marked as materialized, where the result fingerprint includes the query statement feature of the query node, data source version information, and query timestamp; The query results and result fingerprints are stored in the distributed cache system, and a version association relationship is established between the query node and the upstream query node it depends on. The distributed cache system adopts a layered storage architecture, storing frequently accessed query results in the memory layer and infrequently accessed query results in the disk layer. The third unit is used for extracting query statement features in the new query request when a new query request is received, and searching for cache results with the same query statement features in the distributed cache system; If a cached result is found, verify whether the data source version information of the cached result is consistent with the current data source version, and determine whether the dependent data of the cached result has changed based on the version association relationship; When the cache result is verified to be valid, the cache result is returned as a response to a new query request, and the access frequency information of the cache result is updated; when the cache result is verified to be invalid, the query is re-executed and the cache is updated.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images