Data query method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application relate to a data query method and device, electronic equipment and storage medium, the method comprising: obtaining a query request, parsing the query metadata in the query request, and matching the query metadata with a metadata set; determining a target query plan according to the matching result of the query metadata and the metadata set; and executing the target query plan to query data corresponding to the query request. Thus, the delay of data lake data acquisition can be greatly reduced, the performance of data lake analysis is further improved, and real-time data lake analysis is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data query method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rise of the digital wave, more and more enterprises or organizations need to build data lakes for storing massive amounts of data. However, how to achieve efficient data analysis on data lakes has become a problem that plagues the industry.

[0003] Because data lakes are typically built using inexpensive object storage or distributed file systems, and data lake analytics is a typical example of storage-compute separation, data lake I / O (Input / Output) latency is generally quite high, especially during the data acquisition phase, making real-time data lake analytics impossible. Summary of the Invention

[0004] In view of this, in order to solve the technical problem that the latency of data lake IO in the prior art is generally high, especially the latency is very high in the data acquisition stage, thus making it impossible to realize real-time data lake analysis, the embodiments of the present invention provide a data query method, device, electronic device and storage medium.

[0005] In a first aspect, embodiments of the present invention provide a data query method, the method comprising:

[0006] Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset;

[0007] Based on the matching results between the query metadata and the metadata dataset, a target query plan is determined;

[0008] Execute the target query plan to retrieve the data corresponding to the query request.

[0009] In one possible implementation, determining the target query plan based on the matching results between the query metadata and the metadata dataset includes:

[0010] If the matching result is that all the query metadata matches the first metadata in the metadata set, then find the materialized view corresponding to the first metadata;

[0011] The target query plan is determined to be to query the data corresponding to the query request from the materialized view;

[0012] The execution of the target query plan to query the data corresponding to the query request includes:

[0013] Query the data corresponding to the query request from the materialized view.

[0014] In one possible implementation, determining the target query plan based on the matching results between the query metadata and the metadata dataset includes:

[0015] If the matching result is that part of the query metadata in the query metadata matches the second metadata in the metadata set, then find the materialized view corresponding to the second metadata;

[0016] The query request is processed to obtain a first query request and a second query request;

[0017] The target query plan is determined to be to query the first data corresponding to the first query request from the materialized view, and to query the second data corresponding to the second query request from the cache;

[0018] The execution of the target query plan to query the data corresponding to the query request includes:

[0019] Query the first data corresponding to the first query request from the materialized view;

[0020] The second query request is transformed into disjunction normal form, and the target cache block corresponding to the disjunction normal form is searched from the cache.

[0021] Retrieve the second data corresponding to the second query request from the target cache block;

[0022] The union of the first data and the second data is determined to be the data corresponding to the query request.

[0023] In one possible implementation, determining the target query plan based on the matching results between the query metadata and the metadata dataset includes:

[0024] If the matching result is that all the query metadata does not match any metadata in the metadata set, then the target query plan is determined to be to query the data corresponding to the query request from the cache;

[0025] The execution of the target query plan to query the data corresponding to the query request includes:

[0026] Transform the query request into disjunction normal form and search the cache for the target cache block corresponding to the disjunction normal form.

[0027] Retrieve the data corresponding to the query request from the target cache block.

[0028] In one possible implementation, when there are multiple materialized views, determining the target query plan as retrieving data corresponding to the query request from the materialized views includes:

[0029] Calculate the query transformation cost of each materialized view based on the statistical information of each materialized view;

[0030] Based on the query transformation cost, determine the target materialized view from each materialized view;

[0031] The target query plan is determined to be to query the data corresponding to the query request from the target materialized view;

[0032] The step of querying the data corresponding to the query request from the materialized view includes:

[0033] Query the data corresponding to the query request from the target materialized view.

[0034] In one possible implementation, when there are multiple materialized views, determining the target query plan as retrieving first data corresponding to the first query request from the materialized views includes:

[0035] Calculate the query transformation cost of each materialized view based on the statistical information of each materialized view;

[0036] Based on the query transformation cost, determine the target materialized view from each materialized view;

[0037] The target query plan is determined to be to query the first data corresponding to the first query request from the target materialized view;

[0038] The step of querying the first data corresponding to the first query request from the materialized view includes:

[0039] Query the first data corresponding to the first query request from the target materialized view.

[0040] In one possible implementation, the statistical information includes at least one of the following: maximum and minimum values ​​for each column, row average and number of rows, and a histogram of column values;

[0041] The step of calculating the query transformation cost of each materialized view based on the statistical information of each materialized view includes:

[0042] Obtain the weighted sum of the statistical information;

[0043] The weighted sum is determined as the query transformation cost for each materialized view.

[0044] In one possible implementation, the method further includes:

[0045] Receive a materialized view generation request, the materialized view generation request carrying the original data identifier;

[0046] The original data identifier is parsed from the materialized view generation request, and the original data corresponding to the original data identifier is obtained.

[0047] Based on the original data, a materialized view is constructed, and the original data identifier corresponding to the materialized view is stored as metadata in the metadata set;

[0048] Obtain the original data increment corresponding to the materialized view;

[0049] The materialized view is updated based on the increment.

[0050] In one possible implementation, the method further includes:

[0051] Obtain the historical access count of each cache block and the cost of retrieving data stored in each cache block;

[0052] Based on the historical access count and the cost of retrieving stored data, determine the cache blocks to be evicted, and evict the evicted cache blocks from each cache block.

[0053] Secondly, embodiments of the present invention provide a data query device, the device comprising:

[0054] The matching module is used to obtain query requests, parse the query metadata in the query requests, and match the query metadata with the metadata dataset;

[0055] The determination module is used to determine the target query plan based on the matching results between the query metadata and the metadata dataset;

[0056] The execution module is used to execute the target query plan to query the data corresponding to the query request.

[0057] In one possible implementation, the determining module includes:

[0058] The first search unit is configured to search for the materialized view corresponding to the first metadata if the matching result is that all the query metadata matches the first metadata in the metadata set;

[0059] The first determining unit is configured to determine that the target query plan is to query data corresponding to the query request from the materialized view;

[0060] The execution module includes:

[0061] The first query unit is used to query data corresponding to the query request from the materialized view.

[0062] In one possible implementation, the determining module includes:

[0063] The second search unit is used to search for the materialized view corresponding to the second metadata if the matching result is that a portion of the query metadata in the query metadata matches the second metadata in the metadata set;

[0064] The processing unit is used to process the query request to obtain a first query request and a second query request;

[0065] The second determining unit is configured to determine that the target query plan is to query the first data corresponding to the first query request from the materialized view and to query the second data corresponding to the second query request from the cache;

[0066] The execution module includes:

[0067] The second query unit is used to query the first data corresponding to the first query request from the materialized view;

[0068] The third lookup unit is used to transform the second query request into a disjunction paradigm and search for the target cache block corresponding to the disjunction paradigm from the cache.

[0069] The third query unit is used to query the second data corresponding to the second query request from the target cache block;

[0070] The third determining unit is used to determine that the union of the first data and the second data is the data corresponding to the query request.

[0071] In one possible implementation, the determining module includes:

[0072] The fourth determining unit is configured to determine the target query plan as querying the data corresponding to the query request from the cache if the matching result is that all the query metadata does not match any metadata in the metadata set;

[0073] The execution module includes:

[0074] The fourth lookup unit is used to transform the query request into a disjunction paradigm and search for the target cache block corresponding to the disjunction paradigm in the cache.

[0075] The fourth query unit is used to query the data corresponding to the query request from the target cache block.

[0076] In one possible implementation, when there are multiple materialized views, the first determining unit includes:

[0077] The first calculation subunit is used to calculate the query transformation cost of each materialized view based on the statistical information of each materialized view.

[0078] The first determining subunit is used to determine the target materialized view from each materialized view based on the query transformation cost;

[0079] The first query subunit is used to determine that the target query plan is to query the data corresponding to the query request from the target materialized view;

[0080] The first query unit is specifically used for:

[0081] Query the data corresponding to the query request from the target materialized view.

[0082] In one possible implementation, when there are multiple materialized views, the second determining unit includes:

[0083] The second calculation subunit is used to calculate the query transformation cost of each materialized view based on the statistical information of each materialized view.

[0084] The second determining subunit is used to determine the target materialized view from each materialized view based on the query transformation cost;

[0085] The second query subunit is used to determine that the target query plan is to query the first data corresponding to the first query request from the target materialized view;

[0086] The second query unit is specifically used for:

[0087] Query the first data corresponding to the first query request from the target materialized view.

[0088] In one possible implementation, the statistical information includes at least one of the following: the maximum and minimum values ​​of each column, the row average and the number of rows, a histogram of column values, and the first or second calculation subunit, specifically used for:

[0089] Obtain the weighted sum of the statistical information;

[0090] The weighted sum is determined as the query transformation cost for each materialized view.

[0091] In one possible implementation, the device further includes:

[0092] The receiving module is used to receive a materialized view generation request, wherein the materialized view generation request carries the original data identifier;

[0093] The parsing module is used to parse the original data identifier from the materialized view generation request and obtain the original data corresponding to the original data identifier;

[0094] The construction module is used to construct a materialized view based on the original data, and store the original data identifier corresponding to the materialized view as metadata in the metadata set;

[0095] The incremental module is used to obtain the original data increment corresponding to the materialized view;

[0096] An update module is used to update the materialized view based on the increment.

[0097] In one possible implementation, the device further includes:

[0098] The acquisition module is used to obtain the historical access count of each cache block and the cost of acquiring the data stored in each cache block.

[0099] The eviction module is used to determine the cache blocks to be evicted based on the historical access count and the cost of retrieving stored data, and to evict the evicted cache blocks from each cache block.

[0100] Thirdly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the processor is configured to execute a data query program stored in the memory to implement the data query method described in any one of the first aspects.

[0101] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the data query method described in any one aspect.

[0102] The technical solution provided in this invention obtains a query request, parses the query metadata in the query request, matches the query metadata with the metadata dataset, determines a target query plan based on the matching result, and finally executes the target query plan to retrieve the data corresponding to the query request. Since the target query plan is determined based on the metadata matching and then executed, the same execution entity is involved in determining and executing the target query plan. Storage and computation are no longer separated, thus significantly reducing the latency of data lake data acquisition, further improving the performance of data lake analysis, and enabling real-time data lake analysis. Attached Figure Description

[0103] Figure 1 This is a schematic diagram of the architecture of a data query system provided in an embodiment of the present invention;

[0104] Figure 2 A flowchart illustrating an embodiment of a data query method provided by this invention;

[0105] Figure 3 A flowchart illustrating an embodiment of another data query method provided by the present invention;

[0106] Figure 4 A flowchart illustrating another embodiment of the data query method provided by this invention;

[0107] Figure 5 A flowchart illustrating another embodiment of the data query method provided by this invention;

[0108] Figure 6 A flowchart illustrating an embodiment of a materialized view construction and updating method provided by the present invention;

[0109] Figure 7 A flowchart illustrating an embodiment of a cache block eviction method provided by this invention;

[0110] Figure 8 A flowchart illustrating another embodiment of the data query method provided by this invention;

[0111] Figure 9 A block diagram illustrating an embodiment of a data query device provided in this invention;

[0112] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0113] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0114] The following is an exemplary description of the system architecture of the data query method provided in the embodiments of the present invention:

[0115] See Figure 1 This is a schematic diagram of the architecture of the data query system provided in an embodiment of the present invention. Figure 1 As shown, this embodiment of the invention proposes an architecture that combines a real-time OLAP (On-Line Analytical Processing) database and a data lake.

[0116] StarRocks is a new generation of high-speed, full-scenario MPP (Massively Parallel Processing) database, and also a type of OLAP database. This invention uses the StarRocks system to query and analyze data in a data lake, and can fully utilize StarRocks' optimization techniques for OLAP databases, including collocated join, bitmap index, etc.

[0117] StarRocks is primarily divided into two components: FE (Front End) and BE (Back End). The FE is responsible for managing metadata, query planning, query optimization, and scheduling. Figure 1 The coordinating node shown in the unified table view includes the query planner and the metadata engine. The BE (Browser Executioner) is responsible for query execution and data storage. Figure 1 The computing cluster and data lake components.

[0118] A data lake is a centralized repository that allows users to store all structured and unstructured data at any scale. Users can store data as is (without needing to first structure the data) and run different types of analytics, such as from dashboards and visualizations to big data processing, data analysis, and machine learning. Data lakes include, but are not limited to, Hive, Iceberg, Hudi, and Delta Lake.

[0119] The data query method provided by the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0120] See Figure 2 This is a flowchart illustrating an embodiment of a data query method provided by the present invention. Figure 2 As shown, the method may include the following steps:

[0121] Step 201: Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset.

[0122] In this embodiment of the invention, a data query request is obtained, the query metadata in the query request is parsed, and then the query metadata is matched with the metadata dataset.

[0123] based on Figure 1 The system architecture shown and the above-mentioned Figure 1According to the relevant description, after StarRocks obtains a data query request, FE can parse the query metadata in the query request, and then match the query metadata with each metadata in the metadata set stored by FE.

[0124] Step 202: Determine the target query plan based on the matching results between the query metadata and the metadata dataset.

[0125] based on Figure 1 The system architecture shown and the above-mentioned Figure 1 According to the relevant description, FE can use the planner management tool to schedule and optimize queries based on the above matching results, and determine the target query plan.

[0126] As for how the target query plan is determined based on the matching results between the query metadata and the metadata dataset, the following will explain... Figure 3-5 and Figure 8 A detailed description will not be provided here.

[0127] Step 203: Execute the target query plan to retrieve the data corresponding to the query request.

[0128] based on Figure 1 The system architecture shown and the above-mentioned Figure 1 According to the relevant description, BE can execute the target query plan determined by FE to query the data corresponding to the query request.

[0129] This concludes the process. Figure 2 The process described is as follows.

[0130] pass Figure 2 As shown in the flowchart, the technical solution provided by this embodiment of the invention obtains a query request, parses the query metadata in the query request, matches the query metadata with the metadata dataset, determines the target query plan based on the matching result of the query metadata and the metadata dataset, and finally executes the target query plan to query the data corresponding to the query request. Since the target query plan is determined based on the metadata matching and then executed, the same execution entity determines and executes the target query plan, and storage and computation are no longer separated. Therefore, the latency of data lake data acquisition is greatly reduced, further improving the performance of data lake analysis and enabling real-time data lake analysis.

[0131] See Figure 3 This is a flowchart illustrating an embodiment of another data query method provided by the present invention. Figure 3 As shown, the method may include the following steps:

[0132] Step 301: Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset.

[0133] For a detailed description of step 301, please refer to the above. Figure 2 The relevant description of step 201 in the embodiments will not be repeated here.

[0134] Step 302: If the matching result is that all query metadata matches the first metadata in the metadata set, then find the materialized view corresponding to the first metadata.

[0135] The first metadata is the metadata in the metadata set that matches all the aforementioned query metadata. The number of materialized views corresponding to the first metadata can be one or more.

[0136] A materialized view is a database object that includes query results. It is a local copy of remote data or used to generate a summary table based on the sum of data tables. Materialized views store data based on remote tables and can also be called snapshots (similar to snapshots in MSSQL Server, static snapshots).

[0137] based on Figure 1 The system architecture shown and the above-mentioned Figure 1 The relevant description states that if the matching result is that all query metadata matches the first metadata in the FE metadata set, then the materialized view corresponding to the first metadata is searched.

[0138] Many systems have implemented materialized view technologies for data lakes, such as Dremio's Reflection. However, Reflection stores the data alongside the original data, while the materialized view is still built on remote storage. Therefore, it does not address the overhead and latency caused by network access and cannot effectively achieve near-storage computation.

[0139] In this embodiment of the invention, the materialized view is stored in the BE of StarRocks, thereby realizing near-memory computation, reducing network overhead, and improving data query performance.

[0140] As for the specifics of how the materialized view is constructed and updated, the following text will explain. Figure 6 A detailed description will not be provided here.

[0141] Step 303: Determine the target query plan as retrieving the data corresponding to the query request from the materialized view.

[0142] In one embodiment, when there is only one materialized view, the target query plan is determined to be to query the data corresponding to the query request from the materialized view.

[0143] In another embodiment, when there are multiple materialized views, the query transformation cost of each materialized view can be calculated based on the statistical information of each materialized view. Then, based on the query transformation cost, the target materialized view is determined from among the materialized views. Finally, the target query plan is determined to query the data corresponding to the query request from the target materialized view.

[0144] Taking a case where there are three materialized views, namely materialized view A, materialized view B, and materialized view C, the query transformation cost of materialized view A can be calculated based on the statistical information of materialized view A, the query transformation cost of materialized view B can be calculated based on the statistical information of materialized view B, and the query transformation cost of materialized view C can be calculated based on the statistical information of materialized view C.

[0145] From the three materialized views, find the materialized view with the lowest query transformation cost and determine it as the target materialized view. If the query transformation cost of materialized view A < the query transformation cost of materialized view C < the query transformation cost of materialized view B, then materialized view A is determined as the target materialized view.

[0146] The final target query plan was determined to be to retrieve the data corresponding to the query request from materialized view A.

[0147] By calculating the cost of query transformation based on statistical information, the query plan can be optimized. Therefore, data query latency can be further reduced, and data query performance can be improved.

[0148] Optionally, the above statistical information may include at least one of the following: maximum and minimum values ​​for each column, row average and number of rows, and a histogram of column values.

[0149] The specific implementation of calculating the query transformation cost of each materialized view based on the statistical information of each materialized view may include: obtaining the weighted sum of the statistical information, and then determining the weighted sum as the query transformation cost of each materialized view.

[0150] For example, StarRocks will collect statistics such as the maximum and minimum values ​​of each column, the average number of rows and the number of rows, and the histogram of column values ​​for each materialized view. By performing a simple weighted calculation on the statistical data, a value is obtained and used as the query transformation cost for each materialized view.

[0151] It should be noted that the specific statistical data used for weighted calculations to obtain the query transformation cost of each materialized view can be set by the developers according to the actual situation, and this invention does not impose any restrictions on this.

[0152] Step 304: Query the data corresponding to the query request from the materialized view.

[0153] In one embodiment, the BE can query data corresponding to the query request from the materialized view.

[0154] In another embodiment, when there are multiple materialized views, the BE can query the data corresponding to the query request from the target materialized views determined above.

[0155] This concludes the process. Figure 3 The process described is as follows.

[0156] pass Figure 3 As shown in the flowchart, the technical solution provided by this embodiment of the invention obtains a query request, parses the query metadata in the query request, matches the query metadata with the metadata dataset, and then, if the matching result is that all query metadata matches the first metadata in the metadata set, searches for the materialized view corresponding to the first metadata, and determines the target query plan to query the data corresponding to the query request from the materialized view. Finally, the data corresponding to the query request is queried from the materialized view. Since the materialized view is stored in the near end, querying the final data from the materialized view when all query metadata matches the first metadata in the metadata set can reduce network overhead and improve data query performance.

[0157] See Figure 4 This is a flowchart illustrating another embodiment of a data query method provided by the present invention. Figure 4 As shown, the method may include the following steps:

[0158] Step 401: Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset.

[0159] For a detailed description of step 401, please refer to the above. Figure 2 The relevant description of step 201 in the embodiments will not be repeated here.

[0160] Step 402: If the matching result is that part of the query metadata in the query metadata matches the second metadata in the metadata set, then find the materialized view corresponding to the second metadata.

[0161] The second metadata refers to the metadata in the metadata set that matches a portion of the query metadata mentioned above. The number of materialized views corresponding to the second metadata can be one or more, and this embodiment does not limit this.

[0162] based on Figure 1 The system architecture shown indicates that if a portion of the query metadata in the query metadata matches the second metadata in the FE metadata set, the materialized view corresponding to the second metadata is searched.

[0163] Step 403: Process the query requests to obtain the first query request and the second query request.

[0164] Step 404: Determine the target query plan as retrieving the first data corresponding to the first query request from the materialized view, and retrieving the second data corresponding to the second query request from the cache block.

[0165] Steps 403 and 404 will be described in a unified manner below:

[0166] In this embodiment of the invention, only a portion of the query metadata matches the second metadata, but all data needs to be retrieved according to the query request.

[0167] Therefore, the initially obtained query request can be processed to obtain the first query request and the second query request.

[0168] In one embodiment, when there is only one materialized view, the target query plan is determined to be to query the first data corresponding to the first query request from the materialized view and to query the second data corresponding to the second query request from the cache.

[0169] In another embodiment, when there are multiple materialized views, the query transformation cost of each materialized view can be calculated based on the statistical information of each materialized view; based on the query transformation cost, a target materialized view is determined from each materialized view; the target query plan is determined to query the first data corresponding to the first query request from the target materialized view, and to query the second data corresponding to the second query request from the cache.

[0170] Taking a case where there are three materialized views, namely materialized view A, materialized view B, and materialized view C, the query transformation cost of materialized view A can be calculated based on the statistical information of materialized view A, the query transformation cost of materialized view B can be calculated based on the statistical information of materialized view B, and the query transformation cost of materialized view C can be calculated based on the statistical information of materialized view C.

[0171] From the three materialized views, find the materialized view with the lowest query transformation cost and determine it as the target materialized view. If the query transformation cost of materialized view A < the query transformation cost of materialized view C < the query transformation cost of materialized view B, then materialized view A is determined as the target materialized view.

[0172] Finally, the target query plan was determined to be to query the first data that matches the first query request from the materialized view A, and to query the second data that corresponds to the second query request from the cache.

[0173] By calculating the cost of query transformation based on statistical information, the query plan can be optimized. Therefore, data query latency can be further reduced, and data query performance can be improved.

[0174] Optionally, the above statistical information may include at least one of the following: maximum and minimum values ​​for each column, row average and number of rows, and a histogram of column values.

[0175] The specific implementation of calculating the query transformation cost of each materialized view based on the statistical information of each materialized view may include: obtaining the weighted sum of the statistical information, and then determining the weighted sum as the query transformation cost of each materialized view.

[0176] For example, StarRocks will collect statistics such as the maximum and minimum values ​​of each column, the average number of rows and the number of rows, and the histogram of column values ​​for each materialized view. By performing a simple weighted calculation on the statistical data, a value is obtained and used as the query transformation cost for each materialized view.

[0177] It should be noted that the specific statistical data used for weighted calculations to obtain the query transformation cost of each materialized view can be set by the developers according to the actual situation, and this invention does not impose any restrictions on this.

[0178] Step 405: Query the first data corresponding to the first query request from the materialized view.

[0179] In one embodiment, the BE can query first data corresponding to the first query request from the materialized view.

[0180] In another embodiment, when there are multiple materialized views, the first data corresponding to the first query request can be queried from the target materialized views determined above.

[0181] Step 406: Transform the second query request into disjunctive normal form and search for the target cache block corresponding to the disjunctive normal form in the cache.

[0182] In existing technologies, distributed caching is prevalent in most data lake analytics platforms. However, most implementations are simple block caches. These block caches have a relatively large granularity, resulting in more irrelevant content, low utilization, and low query hit rate.

[0183] In the StarRocks provided in this embodiment of the invention, upon receiving a query request, the predicates in the query request are extracted and transformed into disjunctive normal form. Maintaining the disjunctive normal form represents a cache block, and cache blocks can overlap. This improves distributed caching, thereby reducing the granularity of cache blocks, minimizing irrelevant content, and improving query efficiency.

[0184] In one embodiment, the BE can extract the predicate of the second query request and transform the predicate into disjunctive normal form. If all conjunctions of the disjunctive normal form can find a superset in a certain cache block in the cache, then the cache block is determined to have hit the query and is determined to be the target cache block.

[0185] For example, the process of transforming the second query request into disjunctive normal form is as follows:

[0186] Suppose the query is: select t1.a,t1.b,t2.c,t2.d from t1,t2 where(t1.a>1 and t1.b>1 and t2.c>10)

[0187] The predicates corresponding to each table can be transformed into disjunctive normal form, and then the results can be queried and cached.

[0188] SELECT FROM T1 WHERE (t1.a>1 AND t1.b>1) -- Save as RR(T1);

[0189] SELECT FROM T2 WHERE (t2.c>10) -- Save as RR(T2);

[0190] The same query can also use RR(T1) and RR(T2) for the next time. Then, RR(T1) can also be used for the following query:

[0191] select from T1 where(t1.a>1and t1.b>1and t1.c>10).

[0192] Step 407: Query the second data corresponding to the second query request from the target cache block.

[0193] In this embodiment of the invention, BE can query the second data corresponding to the second query request in the target cache block.

[0194] Step 408: Determine the union of the first data and the second data as the data corresponding to the query request.

[0195] In this embodiment of the invention, BE can determine that the union of the first data and the second data is the data corresponding to the query request.

[0196] Furthermore, both materialized view technology and distributed caching technology are transparent to users and do not introduce later operation and maintenance costs.

[0197] This concludes the process. Figure 4 The process described is as follows.

[0198] pass Figure 4 As shown in the flowchart, the technical solution provided in this embodiment of the invention obtains a query request, parses the query metadata in the query request, and matches the query metadata with the metadata dataset. If the matching result is that part of the query metadata in the query metadata matches the second metadata in the metadata set, then the materialized view corresponding to the second metadata is searched. The query request is processed to obtain a first query request and a second query request. The target query plan is determined to be to query the first data corresponding to the first query request from the materialized view, and to query the second data corresponding to the second query request from the cache block. The first data corresponding to the first query request is queried from the materialized view; the second query request is transformed into a disjunctive normal form, and the target cache block corresponding to the disjunctive normal form is searched; the second data corresponding to the second query request is queried from the target cache block; the union of the first data and the second data is determined to be the data corresponding to the query request. By combining distributed caching and materialized view technologies, the latency of data query is greatly reduced, the data analysis capabilities of the data lake are improved, and various data lake analysis scenarios can be optimized.

[0199] See Figure 5 This is a flowchart illustrating another embodiment of the data query method provided by this invention. Figure 5 As shown, the method may include the following steps:

[0200] Step 501: Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset.

[0201] For a detailed description of step 501, please refer to the above. Figure 2 The relevant description of step 201 in the embodiments will not be repeated here.

[0202] Step 502: If the matching result is that all query metadata does not match any metadata in the metadata set, then the target query plan is to query the data corresponding to the query request from the cache.

[0203] In this embodiment of the invention, if all query metadata does not match any metadata in the FE metadata set, it indicates that no corresponding materialized view has been generated in advance, and the target query plan can be determined to be to query the data corresponding to the query request from the cache.

[0204] Step 503: Transform the query request into disjunctive normal form and search the cache for the target cache block that corresponds to the disjunctive normal form.

[0205] Step 504: Retrieve the data corresponding to the query request from the target cache block.

[0206] Steps 503 and 504 will be described in a unified manner below:

[0207] In this embodiment, BE can extract the predicate of the query request and transform the predicate into disjunctive normal form. If all conjunctions of the disjunctive normal form can find a superset in a certain cache block in the cache, then the cache block is determined to have hit the query and is determined to be the target cache block.

[0208] BE can retrieve the data corresponding to the query request from the target cache block.

[0209] This concludes the process. Figure 5 The process described is as follows.

[0210] pass Figure 5 As shown in the flowchart, the technical solution provided in this embodiment of the invention obtains a query request, parses the query metadata in the query request, matches the query metadata with the metadata dataset, and if the matching result is that all query metadata does not match any metadata in the metadata set, then the target query plan is determined to be to query the data corresponding to the query request from the cache, transform the query request into disjunctive normal form, search for the target cache block corresponding to the disjunctive normal form from the cache, and query the data corresponding to the query request from the target cache block. Since in a distributed cache, data retrieval can be completed simply by matching the disjunctive normal form with the cache, the time and space overhead occupied by the index is reduced, and the search efficiency is improved.

[0211] See Figure 6 This is a flowchart illustrating an embodiment of a materialized view construction and update method provided by the present invention. Figure 6 As shown, the method may include the following steps:

[0212] Step 601: Receive the materialized view generation request, which carries the original data identifier.

[0213] In this embodiment of the invention, a user can specify one or more tables stored on the data lake to generate a materialized view as needed, and send a materialized view generation request to StarRocks, which then receives the request. The materialized view generation request carries the original data identifier.

[0214] The aforementioned raw data identifiers may include, but are not limited to: table names, row labels, and / or column labels in the data lake.

[0215] Step 602: Parse the original data identifier from the materialized view generation request and obtain the original data corresponding to the original data identifier.

[0216] Step 603: Based on the original data, construct a materialized view and store the original data identifier corresponding to the materialized view as metadata in the metadata set.

[0217] Steps 602 and 603 will be described in a unified manner below:

[0218] In this embodiment of the invention, StarRocks parses the original data identifier from the materialized view generation request and obtains the original data corresponding to the original data identifier. Then, based on the original data, it constructs the OLAP database of the materialized view itself and stores the original data identifier corresponding to the materialized view as metadata in the metadata set.

[0219] Step 604: Obtain the original data increment corresponding to the materialized view.

[0220] Step 605: Update the materialized view based on the increment.

[0221] Steps 604 and 605 will be described in a unified manner below:

[0222] Most tables on data lakes provide incremental read semantics. StarRocks periodically retrieves the original data increments corresponding to the materialized view and then updates the materialized view based on the increments.

[0223] This concludes the process. Figure 6 The process described is as follows.

[0224] pass Figure 6 As shown in the process, the technical solution provided in this embodiment of the invention receives a materialized view generation request, which carries an original data identifier. The original data identifier is parsed from the materialized view generation request to obtain the original data corresponding to the original data identifier. Based on the original data, a materialized view is constructed, and the original data identifier corresponding to the materialized view is stored as metadata in a metadata set. The incremental value of the original data corresponding to the materialized view is obtained. The materialized view is updated based on the incremental value. Since the user specifies the materialization process by writing materialized view SQL, StarRocks automatically creates the materialized view and continuously updates it incrementally. Furthermore, because the construction of materialized views is usually completed automatically in the background, it can proactively accelerate the process without relying on user-initiated queries. Simultaneously, by continuously obtaining incremental values ​​to update the materialized view, the purpose of real-time maintenance of the materialized view is achieved.

[0225] See Figure 7 This is a flowchart illustrating an embodiment of a cache block eviction method provided by this invention. Figure 7 As shown, the method may include the following steps:

[0226] Step 701: Obtain the historical access count of each cache block and the cost of retrieving data stored in each cache block.

[0227] Step 702: Based on the historical access count and the cost of retrieving stored data, determine the cache blocks to be evicted, and evict the evicted cache blocks from each cache block.

[0228] Steps 701 and 702 are described in a unified manner below:

[0229] In this embodiment of the invention, the historical access counts of each cache block and the cost of retrieving data stored in each cache block can be obtained. Then, based on the historical access counts and the cost of retrieving stored data, cache blocks to be evicted are determined, and evicted cache blocks are removed from each cache block.

[0230] StarRocks analyzes user query requests and relocates frequently accessed cached data closer to compute nodes, such as their local disks or memory. This approach is seamless for users and requires no manual maintenance, but it suffers from some latency. Therefore, StarRocks employs a cache eviction policy to remove cached blocks.

[0231] StarRocks' cache eviction policy is a cost-based LRU (Least Recently Used) policy. Unlike the general LRU, StarRocks not only counts the historical access count of each cache block, but also records the cost of acquiring each cache block.

[0232] Generally, within a certain period, cache blocks with a historical access count lower than a preset value are identified as cache blocks to be evicted, and these evicted cache blocks are removed from the cache. Given the same historical access count, cache blocks with higher retrieval costs are more likely to remain in the cache; that is, cache blocks with lower retrieval costs are identified as cache blocks to be evicted, and these evicted cache blocks are removed from the cache.

[0233] This concludes the process. Figure 7 The process described is as follows.

[0234] pass Figure 7 As shown in the flowchart, the technical solution provided by this embodiment of the invention obtains the historical access counts of each cache block and the acquisition cost of the data stored in each cache block. Based on these historical access counts and acquisition costs, it determines the cache blocks to be evicted and evicts them from each cache block. This optimizes the distributed cache, improves the hit rate of user data queries, further enhances the performance of data lake analysis, and facilitates real-time data lake analysis.

[0235] See Figure 8 The following is a flowchart illustrating another embodiment of a data query method provided by this invention. Figure 8As shown, the method may include the following steps:

[0236] Step 801: Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset.

[0237] Step 802: Determine the target query plan based on the matching results between the query metadata and the metadata set. If the matching result is that all query metadata matches the first metadata in the metadata set, proceed to step 803; if the matching result is that some query metadata in the query metadata matches the second metadata in the metadata set, proceed to step 806; if the matching result is that all query metadata does not match any metadata in the metadata set, proceed to step 813.

[0238] Step 803: Locate the materialized view corresponding to the first metadata.

[0239] Step 804: Determine the target query plan as retrieving the data corresponding to the query request from the materialized view.

[0240] Step 805: Query the data corresponding to the query request from the materialized view.

[0241] Step 806: Locate the materialized view corresponding to the second metadata.

[0242] Step 807: Process the query requests to obtain the first query request and the second query request.

[0243] Step 808: Determine the target query plan as retrieving the first data corresponding to the first query request from the materialized view, and retrieving the second data corresponding to the second query request from the cache block.

[0244] Step 809: Query the first data corresponding to the first query request from the materialized view.

[0245] Step 810: Transform the second query request into disjunctive normal form and search for the target cache block corresponding to the disjunctive normal form in the cache.

[0246] Step 811: Query the second data corresponding to the second query request from the target cache block.

[0247] Step 812: Determine the union of the first data and the second data as the data corresponding to the query request.

[0248] Step 813: Determine the target query plan as retrieving the data corresponding to the query request from the cache.

[0249] Step 814: Transform the query request into disjunctive normal form and search the cache for the target cache block that corresponds to the disjunctive normal form.

[0250] Step 815: Retrieve the data corresponding to the query request from the target cache block.

[0251] For a detailed description of steps 801-815, please refer to the above. Figure 2-5 The relevant descriptions in the embodiments will not be repeated here.

[0252] Corresponding to the aforementioned embodiments of the data query method, the present invention also provides embodiments of a data query device.

[0253] See Figure 9 This is a block diagram of an embodiment of a data query device provided by an embodiment of the present invention.

[0254] like Figure 9 As shown, the device includes:

[0255] The matching module 901 is used to obtain a query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset;

[0256] The determination module 902 is used to determine the target query plan based on the matching results between the query metadata and the metadata dataset;

[0257] The execution module 903 is used to execute the target query plan to query the data corresponding to the query request.

[0258] In one possible implementation, the determining module 902 includes (not shown in the figure):

[0259] The first search unit is configured to search for the materialized view corresponding to the first metadata if the matching result is that all the query metadata matches the first metadata in the metadata set;

[0260] The first determining unit is configured to determine that the target query plan is to query data corresponding to the query request from the materialized view;

[0261] The execution module 903 includes (not shown in the figure):

[0262] The first query unit is used to query data corresponding to the query request from the materialized view.

[0263] In one possible implementation, the determining module 902 includes (not shown in the figure):

[0264] The second search unit is used to search for the materialized view corresponding to the second metadata if the matching result is that a portion of the query metadata in the query metadata matches the second metadata in the metadata set;

[0265] The processing unit is used to process the query request to obtain a first query request and a second query request;

[0266] The second determining unit is configured to determine that the target query plan is to query the first data corresponding to the first query request from the materialized view and to query the second data corresponding to the second query request from the cache;

[0267] The execution module 903 includes (not shown in the figure):

[0268] The second query unit is used to query the first data corresponding to the first query request from the materialized view;

[0269] The third lookup unit is used to transform the second query request into a disjunction paradigm and search for the target cache block corresponding to the disjunction paradigm from the cache.

[0270] The third query unit is used to query the second data corresponding to the second query request from the target cache block;

[0271] The third determining unit is used to determine that the union of the first data and the second data is the data corresponding to the query request.

[0272] In one possible implementation, the determining module 902 includes (not shown in the figure):

[0273] The fourth determining unit is configured to determine the target query plan as querying the data corresponding to the query request from the cache if the matching result is that all the query metadata does not match any metadata in the metadata set;

[0274] The execution module 903 includes (not shown in the figure):

[0275] The fourth lookup unit is used to transform the query request into a disjunction paradigm and search for the target cache block corresponding to the disjunction paradigm in the cache.

[0276] The fourth query unit is used to query the data corresponding to the query request from the target cache block.

[0277] In one possible implementation, when there are multiple materialized views, the first determining unit includes (not shown in the figures):

[0278] The first calculation subunit is used to calculate the query transformation cost of each materialized view based on the statistical information of each materialized view.

[0279] The first determining subunit is used to determine the target materialized view from each materialized view based on the query transformation cost;

[0280] The first query subunit is used to determine that the target query plan is to query the data corresponding to the query request from the target materialized view;

[0281] The first query unit is specifically used for:

[0282] Query the data corresponding to the query request from the target materialized view.

[0283] In one possible implementation, when there are multiple materialized views, the second determining unit includes (not shown in the figure):

[0284] The second calculation subunit is used to calculate the query transformation cost of each materialized view based on the statistical information of each materialized view.

[0285] The second determining subunit is used to determine the target materialized view from each materialized view based on the query transformation cost;

[0286] The second query subunit is used to determine that the target query plan is to query the first data corresponding to the first query request from the target materialized view;

[0287] The second query unit is specifically used for:

[0288] Query the first data corresponding to the first query request from the target materialized view.

[0289] In one possible implementation, the statistical information includes at least one of the following: maximum and minimum values ​​for each column, row average and number of rows, and a histogram of column values;

[0290] The first or second computing subunit is specifically used for:

[0291] Obtain the weighted sum of the statistical information;

[0292] The weighted sum is determined as the query transformation cost for each materialized view.

[0293] In one possible implementation, the device further includes (not shown in the figures):

[0294] The receiving module is used to receive a materialized view generation request, wherein the materialized view generation request carries the original data identifier;

[0295] The parsing module is used to parse the original data identifier from the materialized view generation request and obtain the original data corresponding to the original data identifier;

[0296] The construction module is used to construct a materialized view based on the original data, and store the original data identifier corresponding to the materialized view as metadata in the metadata set;

[0297] The incremental module is used to obtain the original data increment corresponding to the materialized view;

[0298] An update module is used to update the materialized view based on the increment.

[0299] In one possible implementation, the device further includes (not shown in the figures):

[0300] The acquisition module is used to obtain the historical access count of each cache block and the cost of acquiring the data stored in each cache block.

[0301] The eviction module is used to determine the cache blocks to be evicted based on the historical access count and the cost of retrieving stored data, and to evict the evicted cache blocks from each cache block.

[0302] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 10 The illustrated electronic device 1000 includes at least one processor 1001, a memory 1002, at least one network interface 1004, and a user interface 1003. The various components in the electronic device 1000 are coupled together via a bus system 1005. It is understood that the bus system 1005 is used to implement communication between these components. In addition to a data bus, the bus system 1005 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 10 The general labeled all buses as Bus System 1005.

[0303] The user interface 1003 may include a display, keyboard or clicking device (e.g., mouse, trackball), touchpad or touch screen, etc.

[0304] It is understood that the memory 1002 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1002 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0305] In some implementations, memory 1002 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 10021 and application programs 10022.

[0306] The operating system 10021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 10022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 10022.

[0307] In this embodiment of the invention, by calling the program or instructions stored in the memory 1002, specifically the program or instructions stored in the application program 10022, the processor 1001 executes the method steps provided in each method embodiment, including, for example:

[0308] Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset;

[0309] Based on the matching results between the query metadata and the metadata dataset, a target query plan is determined;

[0310] Execute the target query plan to retrieve the data corresponding to the query request.

[0311] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 1001. Processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1002. Processor 1001 reads the information in memory 1002 and, in conjunction with its hardware, completes the steps of the above method.

[0312] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0313] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0314] The electronic device provided in this embodiment may be as follows: Figure 10 The electronic device shown can perform the following: Figure 2-8 All steps of the data query method in the middle, thereby realizing Figure 2-8 For details on the technical effectiveness of the data query method shown, please refer to [link / reference]. Figure 2-8 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0315] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0316] When one or more programs in the storage medium can be executed by one or more processors to implement the data query method described above that is executed on the electronic device side.

[0317] The processor is used to execute a data query program stored in the memory to implement the following steps of a data query method executed on the electronic device side:

[0318] Obtain the query request, parse the query metadata in the query request, and match the query metadata with the metadata dataset;

[0319] Based on the matching results between the query metadata and the metadata dataset, a target query plan is determined;

[0320] Execute the target query plan to retrieve the data corresponding to the query request.

[0321] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0322] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0323] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data query method, characterized by, The method comprises: acquiring a query request, parsing query metadata in the query request, and matching the query metadata with a metadata set; determining a target query plan according to a matching result of the query metadata and the metadata set; executing the target query plan to query data corresponding to the query request; wherein the determining the target query plan according to the matching result of the query metadata and the metadata set comprises: if the matching result is that part of the query metadata matches second metadata in the metadata set, finding a materialized view corresponding to the second metadata; processing the query request to obtain a first query request and a second query request; determining the target query plan as querying first data corresponding to the first query request from the materialized view and querying second data corresponding to the second query request from a cache; the executing the target query plan to query data corresponding to the query request comprises: querying the first data corresponding to the first query request from the materialized view; transforming the second query request into a disjunctive normal form, finding a target cache block corresponding to the disjunctive normal form from the cache; querying the second data corresponding to the second query request from the target cache block; determining a union set of the first data and the second data as data corresponding to the query request; in a case where the number of the materialized views is multiple, the determining the target query plan as querying the first data corresponding to the first query request from the materialized view comprises: calculating query transformation costs of each materialized view according to statistical information of each materialized view; determining a target materialized view from each materialized view according to the query transformation costs; determining the target query plan as querying the first data corresponding to the first query request from the target materialized view; the querying the first data corresponding to the first query request from the materialized view comprises: querying the first data corresponding to the first query request from the target materialized view.

2. The method of claim 1, wherein, the determining the target query plan according to the matching result of the query metadata and the metadata set comprises: if the matching result is that all the query metadata matches first metadata in the metadata set, finding a materialized view corresponding to the first metadata; determining the target query plan as querying data corresponding to the query request from the materialized view; the executing the target query plan to query data corresponding to the query request comprises: querying data corresponding to the query request from the materialized view.

3. The method of claim 1, wherein, the determining the target query plan according to the matching result of the query metadata and the metadata set comprises: if the matching result is that all the query metadata does not match any metadata in the metadata set, determining the target query plan as querying data corresponding to the query request from a cache; the executing the target query plan to query data corresponding to the query request comprises: transform the query request into a disjunctive normal form, find a target cache block corresponding to the disjunctive normal form from the cache; query data corresponding to the query request from the target cache block.

4. The method of claim 2, wherein, In a case where the number of the materialized views is multiple, the determining the target query plan to query data corresponding to the query request from the materialized view comprises: calculating query transformation costs of the materialized views according to statistical information of the materialized views; determining a target materialized view from the materialized views according to the query transformation costs; determining the target query plan to query data corresponding to the query request from the target materialized view; the querying data corresponding to the query request from the materialized view comprises: querying data corresponding to the query request from the target materialized view.

5. The method according to any of claims 1 or 4, characterized in that, The statistical information at least comprises one of the following: maximum value, minimum value, row average value and row number of each column, and histogram of column values; the calculating query transformation costs of the materialized views according to the statistical information of the materialized views comprises: obtaining a weighted sum of the statistical information; determining the weighted sum as the query transformation cost of the materialized view.

6. The method of any of claims 1 or 4, wherein, The method further comprises: receiving a materialized view generation request, the materialized view generation request carrying original data identification; parsing the original data identification from the materialized view generation request, obtaining original data corresponding to the original data identification; constructing a materialized view according to the original data, and storing original data identification corresponding to the materialized view as metadata in the metadata set; obtaining an original data increment corresponding to the materialized view; updating the materialized view according to the increment.

7. The method of any of claims 1 or 3, wherein, The method further comprises: obtaining historical access times of each cache block and obtaining costs of stored data in each cache block; determining an eliminated cache block according to the historical access times and the obtaining costs of the stored data, and eliminating the eliminated cache block from each cache block.

8. A data query apparatus, characterized by comprising: The apparatus comprises: a matching module configured to obtain a query request, parse query metadata in the query request, and match the query metadata with a metadata set; a determining module configured to determine a target query plan according to a matching result of the query metadata and the metadata set; an executing module configured to execute the target query plan to query data corresponding to the query request; The determining module is specifically configured to: if the matching result is that part of the query metadata matches second metadata in the metadata set, find a materialized view corresponding to the second metadata; process the query request to obtain a first query request and a second query request; determine the target query plan to query first data corresponding to the first query request from the materialized view, and query second data corresponding to the second query request from a cache; The executing module is specifically configured to: query the first data corresponding to the first query request from the materialized view; transform the second query request into a disjunctive normal form, find a target cache block corresponding to the disjunctive normal form from the cache; query second data corresponding to the second query request from the target cache block; determine a union of the first data and the second data as data corresponding to the query request; in a case where the number of the materialized views is multiple, the determining module is further configured to: calculate a query transformation cost of each materialized view according to statistical information of each materialized view; determine a target materialized view from each materialized view according to the query transformation cost; determine the target query plan as querying first data corresponding to the first query request from the target materialized view; the executing module is further configured to: query first data corresponding to the first query request from the target materialized view.

9. An electronic device, comprising: comprise: a processor and a memory, the processor is used to execute a data query program stored in the memory, so as to realize the data query method in any one of claims 1-7.

10. A storage medium, characterized by The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the data query method in any one of claims 1-7.

Images