Data processing method, device, equipment, medium and program product

Abstract

The present invention provides a data processing method, device, equipment, medium and program product, which can be applied to the field of database storage management. The method includes: parsing a query statement to obtain a query plan tree; executing a query plan of the query plan tree to obtain multiple data blocks; establishing a directed topology graph according to the data blocks and the execution logic relationship between the data blocks; querying a target data block having an execution logic relationship with a cache data block from the directed topology graph, wherein the cache data block is stored in a cache associated with the database; and storing the target data block in the cache associated with the database.

Description

Technical Field

[0001] The present invention relates to the field of database storage management, and more specifically to a data processing method, device, equipment, medium and program product. Background Art

[0002] In order to improve the efficiency of data access, database management systems have established powerful data cache areas to obtain required data from the cache and reduce the number of disk accesses, thereby increasing the speed of data reading and improving database query performance. The cache needs to pre-fetch data before the query accesses the data, otherwise the query needs to hang and wait for the cache to load the data from the disk to the memory. Most database cache replacement strategies use the traditional least recently used algorithm as a cache replacement mechanism, but when the amount of data in a single table is much larger than the memory, the cache hit rate will be extremely low. At this time, the role of the cache is greatly reduced. For the reading of super large tables, data is read directly from the disk file to avoid using the cache. In this case, the core role of the cache replacement strategy is reduced, and the pre-fetch strategy is increased. Summary of the invention

[0003] In view of the above problems, the present invention provides a data processing method, apparatus, device, medium and program product.

[0004] According to a first aspect of the present invention, there is provided a data processing method, comprising: parsing a query statement to obtain a query plan tree; executing a query plan of the query plan tree to obtain a plurality of data blocks; establishing a directed topological graph according to the data blocks and the execution logical relationships between the data blocks; querying a target data block having an execution logical relationship with a cache data block from the directed topological graph, wherein the cache data block is stored in a cache associated with a database; and storing the target data block in a cache associated with the database.

[0005] Optionally, executing the query plan of the query plan tree to obtain multiple data blocks includes: determining a common subtree from multiple query plan trees; and executing the query plan of the common subtree to obtain multiple data blocks.

[0006] Optionally, determining a common subtree from multiple query plan trees includes: matching a reference query subtree in a reference query plan tree with a benchmark query subtree in a benchmark query plan tree according to a matching rule to obtain a matching result, wherein the multiple query plan trees include a reference query plan tree and a benchmark query plan tree; when the matching result indicates a match, determining the reference query subtree or the benchmark query subtree as a common subtree.

[0007] Optionally, the matching rule represents that: the connection conditions of the base query subtree and the reference query subtree are the same; and the query subnode of the base query subtree and the query subnode of the reference query subtree are the same.

[0008] Optionally, querying from the directed topological graph for a target data block having an execution logical relationship with the cached data block includes: searching from the directed topological graph according to the cached data block to obtain multiple connected data blocks, the cached data block and the connected data block having an execution logical relationship; determining the in-degree value of the connected data block in the directed topological graph according to the execution logical relationship between the data blocks; and determining the target data block according to the in-degree value of the connected data block.

[0009] Optionally, determining the target data block according to the in-degree values ​​of the connected data blocks includes: obtaining a target query statement when the in-degree values ​​of each of the multiple connected data blocks satisfy a preset in-degree value condition; parsing the target query statement to obtain a target query plan tree; executing the query plan of the target query plan tree to obtain an intermediate execution data block; and determining a connected data block among the multiple connected data blocks that matches the intermediate execution data block as the target data block.

[0010] A second aspect of the present invention provides a data processing device, comprising:

[0011] The parsing module is used to parse the query statement and obtain the query plan tree;

[0012] A data block obtaining module is used to execute a query plan of a query plan tree to obtain multiple data blocks;

[0013] A directed topology graph building module is used to build a directed topology graph according to data blocks and execution logic relationships between data blocks;

[0014] a query module, configured to query a target data block having an execution logic relationship with a cache data block from the directed topology graph, wherein the cache data block is stored in a cache associated with a database; and

[0015] The storage module is used to store the target data block in a cache associated with the database.

[0016] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0017] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0018] The fifth aspect of the present invention also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0019] Optionally, by parsing the query statement, a query plan tree is obtained, and the query plan of the query plan tree is executed to obtain multiple data blocks. A directed topology graph is established based on the data blocks and the connection relationship between the data blocks. Finally, the next data block to be pre-fetched is searched from the directed topology graph based on the cache data blocks existing in the cache. In the pre-fetching process, a directed topology graph is established by considering the relationship between the query statement and the data block. The directed topology graph can be used to predict the data blocks that the query statement will read in the future, thereby strengthening the data cache pre-fetching in the cache management, reducing the cache miss rate, and improving the cache hit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0021] Figure 1 A diagram schematically shows an application scenario of a data processing method according to an embodiment of the present invention;

[0022] Figure 2 A flowchart of a data processing method according to an embodiment of the present invention is schematically shown;

[0023] Figure 3A-3B A schematic diagram of extracting a common subtree according to an embodiment of the present invention is schematically shown;

[0024] Figure 4A-4B Schematically shows a data block relationship network diagram according to an embodiment of the present invention;

[0025] Figure 5 A structural block diagram of a data processing device according to an embodiment of the present invention is schematically shown;

[0026] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0027] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] When using a database to query table data, it is necessary to access the table data. To obtain data from the data table, it sends a request to the storage manager to obtain data. The access to the table is very high frequency. In order to improve the efficiency of data access, modern database management systems have established a powerful data cache area to obtain the required data from the cache and reduce the number of disk accesses, thereby increasing the speed of data reading and improving database query performance. Data experiments have shown that using memory cache to read data is at least two orders of magnitude faster than directly reading and writing mechanical hard disks without using cache. This poses a problem. The cache needs to pre-fetch data before the query accesses the data, otherwise the query needs to hang and wait for the cache to load the data from the disk to the memory. Most modern database cache replacement strategies use the traditional Least Recently Used (LRU) algorithm as a cache replacement mechanism, at least for Poisson databases PostgreSQL and MySQL. When dealing with traditional table-based statistics and access information, it can maintain good performance. However, under analytical data queries, these traditional cache replacement mechanisms are no longer effective. Because the amount of data in a single table is much larger than the memory, the cache hit rate is extremely low. At this time, the role of the cache is greatly reduced. For reading super large tables, data is directly read from disk files to avoid using cache. In this case, the core role of the cache is reduced in replacement strategy and increased in prefetch strategy. Currently, prefetching basically relies on the prefetching function of hardware.

[0032] The basic logic of data prefetching is as follows: when the data queryer obtains the first batch of data, it notifies the cache manager to load the second batch of data into the cache in advance. When the data queryer obtains the second batch of data, it notifies the cache manager to load the third batch of data in advance, and the first batch of data can be removed from the cache.

[0033] In most cases, the data queryer does not know what data is needed next, or the database does not provide a function to specify which data to pre-fetch. Instead, the database provides a random pre-fetch function (for example, after querying data 1, 2, 3, it thinks you may also need 7, 8, 9, so it loads 7, 8, 9 into the cache in advance) or a sequential cache function (after executing a query, other data adjacent to the query data on the disk is also pre-fetched into the cache). This situation will cause cache pollution, resulting in a decrease in the performance gain of replacing the cache.

[0034] The goal of cache prefetching is to enable the cache manager prefetching module to predict the data blocks that will be read in the future by the Structured Query Language (SQL), thereby strengthening data cache prefetching in cache management, reducing cache miss rates, and improving cache hit rates. In the past, there was little work on data prefetching, and priority was given to scheduling those data blocks into the data cache. More work is on studying which data blocks should be retained in the cache. There has been some progress in the research on cache prefetching of the CPU, which can be provided to the data block system as a reference for improvement, but these have not taken into account that the database itself is an operation of the SQL programming language, and more than 90% of the queries in analytical databases are fixed. Even the data is updated in batches.

[0035] In the process of implementing the present invention, it is found that there are two problems with cache prefetching management in current database management systems: (1) low cache hit rate when dealing with large amounts of data; and (2) cache prefetching relies on hardware cache without considering the relationship between SQL and data blocks.

[0036] In view of this, the present invention provides a data processing method, device, electronic device and medium, the method comprising: parsing a query statement to obtain a query plan tree; executing a query plan of the query plan tree to obtain multiple data blocks; establishing a directed topological graph based on the data blocks and the execution logical relationship between the data blocks; querying a target data block that has an execution logical relationship with a cache data block from the directed topological graph, wherein the cache data block is stored in a cache associated with a database; and storing the target data block in a cache associated with the database.

[0037] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] Figure 1 The application scenario diagram of the data processing method according to an embodiment of the present invention is schematically shown.

[0039] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] The following will be based on Figure 1 The scene described by Figure 2 , 3A , 3B, 4A and Figure 4B The data processing method of the disclosed embodiment is described in detail.

[0046] Figure 2 The flowchart of the data processing method according to the embodiment of the present invention is schematically shown.

[0047] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S250.

[0048] In operation S210, the query statement is parsed to obtain a query plan tree.

[0049] In operation S220 , the query plan of the query plan tree is executed to obtain a plurality of data blocks.

[0050] In operation S230, a directed topology graph is established according to the data blocks and the execution logic relationships between the data blocks.

[0051] In operation S240 , a target data block having an execution logic relationship with the cache data block is searched from the directed topology graph.

[0052] In operation S250 , the target data block is stored in a cache associated with the database.

[0053] Optionally, a query statement can be used to retrieve data that meets the conditions from the database. The query data source can be from one table or from multiple tables. The query result can be a record set consisting of 0 or more rows of records, and one or more fields can be selected as output fields.

[0054] Optionally, parsing the query statement may obtain a query plan tree, which may represent a tree representation of an abstract syntax structure of the source code, wherein each node on the tree identifies a structure in the source code.

[0055] Optionally, each SQL query is represented as a query plan with a tree structure. In order to reduce the number of query plan trees, the query plan recommended by the optimizer can be used. The query plan recommended by the optimizer is more likely to contain high-yield materialized views. The physical query plan with a tree structure is extracted from the optimizer, and on this basis, common subqueries can be further extracted.

[0056] Optionally, receive the query statement and parse it to get the query plan tree. The parser supports syntax extension to support the parsing needs of different syntaxes of multiple data sources such as Mysql and Hive. For example, the open source syntax parsers antlr and javacc can support multiple data source syntax by extending the syntax rule file. Calcite and Druid have built-in programmatic syntax parsers. After the query statement passed by the user is parsed by the parser, all nodes are traversed.

[0057] Optionally, the query plan of the query plan tree can be executed to obtain multiple data blocks. Each SQL subquery representation will be turned into an execution plan for final execution. During the execution process, the required data blocks will be retrieved from the disk one by one.

[0058] Optionally, a directed topology graph can be established based on the data blocks and the execution logic relationship between the data blocks, that is, the front-back relationship of the data blocks. Each time an edge is added between two data block nodes, the more edges there are between the two nodes, the greater the weight and the closer the relationship.

[0059] Optionally, after obtaining the directed topology graph, the target data block having an execution logic relationship with the cache data block can be queried from the directed topology graph, and the cache data block is stored in a cache associated with the database. For example, if the directed topology graph is 1—>2—>5—>4—>7, and the cache data block is 5, the target data block 4 having an execution logic relationship with the cache data block 5 can be queried from the directed topology graph.

[0060] Optionally, the obtained target data block may be stored in a cache associated with the database.

[0061] Optionally, by parsing the query statement, a query plan tree is obtained, and the query plan of the query plan tree is executed to obtain multiple data blocks. A directed topology graph is established based on the data blocks and the connection relationship between the data blocks. Finally, the next data block to be pre-fetched is searched from the directed topology graph based on the cache data blocks existing in the cache. In the pre-fetching process, a directed topology graph is established by considering the relationship between the query statement and the data block. The directed topology graph can be used to predict the data blocks that the query statement will read in the future, thereby strengthening the data cache pre-fetching in the cache management, reducing the cache miss rate, and improving the cache hit rate.

[0062] Optionally, executing the query plan of the query plan tree to obtain multiple data blocks includes: determining a common subtree from multiple query plan trees; and executing the query plan of the common subtree to obtain multiple data blocks.

[0063] Optionally, a common subtree may be determined from multiple query plan trees, and the query plan of the common subtree may be executed to obtain multiple data blocks.

[0064] Optionally, determining the common subtree can infer that the data block with the highest cache pre-fetch probability enters the cache, thereby improving the cache hit rate.

[0065] Optionally, determining a common subtree from multiple query plan trees includes: matching a reference query subtree in a reference query plan tree with a benchmark query subtree in a benchmark query plan tree according to a matching rule to obtain a matching result, wherein the multiple query plan trees include a reference query plan tree and a benchmark query plan tree; when the matching result indicates a match, determining the reference query subtree or the benchmark query subtree as a common subtree.

[0066] Optionally, a common subtree can be determined from multiple query plan trees. According to a matching rule, a reference query subtree in a reference query plan tree can be matched with a benchmark query subtree in a benchmark query plan tree to obtain a matching result. The multiple query plan trees include a reference query plan tree and a benchmark query plan tree. When the matching result indicates a match, the reference query subtree or the benchmark query subtree is determined as a common subtree.

[0067] Optionally, a common subtree may correspond to a corresponding SQL subquery statement. After the common subtree may be determined, the query plan of the common subtree may be executed to obtain multiple data blocks. A directed topology graph is established based on the data blocks and the execution logic relationship between the data blocks, and a target data block having an execution logic relationship with the cache data block is queried from the directed topology graph. After the target data block is obtained, the target data block is stored in a cache associated with the database.

[0068] Optionally, by considering the connection between SQL query statements and data blocks, the cache hit rate is improved to support dynamic changes in SQL and data.

[0069] Optionally, the matching rule represents that: the connection conditions of the base query subtree and the reference query subtree are the same; and the query subnode of the base query subtree and the query subnode of the reference query subtree are the same.

[0070] Optionally, the matching rule may characterize that the connection conditions of the base query subtree and the reference query subtree are the same and the query subnodes of the base query subtree and the query subnodes of the reference query subtree are the same. When detecting a common subtree, if the base query subtree and the reference query subtree have similar connection, selection or projection conditions, and the subnodes of the base query subtree and the reference query subtree are equivalent, then the base query subtree or the base query subtrees are equivalent and can be determined to be a common subtree.

[0071] Optionally, all query plan trees in the workload are merged into a directed topology graph by calculating the benefit of each common subtree. In a directed topology graph, equivalent subtrees with the same structure are merged into a common subtree, which makes it easy to find the common subtree. By merging equivalent subtrees in a bottom-up manner and iteratively merging query statements into a directed topology graph, the frequency of each node in the query graph can be calculated.

[0072] Figure 3A-3B The schematic diagram of extracting a common subtree according to an embodiment of the present invention is schematically shown.

[0073] like Figure 3A As shown in the figure, q1, q2, and q3 are different query statements. After parsing q1, q2, and q3, we can obtain the query plan trees Q1, Q2, and Q3 corresponding to q1, q2, and q3 respectively. Determine the common subtree from the query plan trees Q1, Q2, and Q3. For example, the query statement for querying the nickname field and email field of the person named "Zhang San" in the testtable table is SELECT nickname FROMtestable WHERE name='Zhang San'. Among them, (A, a) can be expressed as the original query statement SELECTnickname. It can be expressed as FROM testable. It can be represented as the value 'Zhang San' determined on the subtree.

[0074] Optionally, the query plan tree Q1 can be used as the base query tree, and the query plan trees Q2 and Q3 can be used as reference query trees respectively. According to the matching rules, it is determined whether the base query subtree in the base query tree and the reference query subtree in the reference query tree have the same connection conditions. If they have the same connection conditions, it is determined whether the values ​​of the child nodes are the same. It can be represented as the value determined on the subtree. If the values ​​in the child nodes are also the same, the reference query subtree or the base query subtree is determined as a common subtree. Among them, A, B, C, D, E, F, and G can all be represented as the reference query subtree or base query subtree finally determined on this branch in the query plan tree. Figure 3A The reference query subtree 31 in the query tree can determine that the same connection condition is found in the reference query tree Q2 and the reference query tree Q3, but because the value a='20' on the subtree 31 is different from the value a='xxx' on the subtree 34 and the value a='250' on the subtree 37, they have no common subtree. Figure 3A The reference query subtree 32 in can find the reference query subtree 35 with the same connection condition on the reference query tree Q2. Because the value b='abc' on the reference query subtree 32 is the same as the value b='abc' on the reference query subtree 35, the reference query subtree 32 or the reference query subtree 35 can be considered as a common subtree. Figure 3A The benchmark query subtree 33 can find subtree 36 and subtree 38 with the same connection conditions in reference query tree Q2 and reference query tree Q3, but because the value d='250' on the benchmark query subtree 33 is the same as the value d='250' on the reference query subtree 38, the benchmark query subtree 33 or the reference query subtree 38 can be considered as a common subtree.

[0075] like Figure 3B As shown by Figure 3A The corresponding common subtree can be found. The reference query subtree 32 is a common subtree of the reference query tree Q1 and the reference query tree Q2. V1 can be marked as the frequency of occurrence of the common subtree. Figure 3B B, C, D, and E below can be represented as the subtrees finally determined on this branch in the query plan tree. It can be represented as a value determined on a subtree. The reference query subtree 36 is a subtree with the same connection condition shared by the query plan trees Q1, Q2 and Q3, and the base query subtree 33 is a common subtree of the base query tree Q1 and the reference query tree Q3. The frequencies V2 and V3 of the common subtrees can be marked on the common subtrees as the number of occurrences.

[0076] Optionally, querying from the directed topological graph for a target data block that has an execution logical relationship with the cached data block includes: searching from the directed topological graph based on the cached data block to obtain multiple connected data blocks, the cached data block and the connected data block having an execution logical relationship; determining the in-degree value of the connected data block in the directed topological graph based on the execution logical relationship between the data blocks; and determining the target data block based on the in-degree value of the connected data block.

[0077] Optionally, a search can be performed from a directed topology graph based on the cache data block to obtain multiple connection data blocks. The cache data block has an execution logic relationship with the connection data block. The in-degree value of the connection data block in the directed topology graph can be determined based on the execution logic relationship between the data blocks. Finally, the target data block is determined based on the in-degree value of the connection data block. For example, the cache data block is 3, and a search is performed from the directed topology graph. There are multiple connection data blocks that have an execution logic relationship with the cache data block 3, which are 5, 7, and 8. The in-degree values ​​of the connection data blocks are determined from the directed topology graph, which are 1, 2, and 3 respectively. According to the in-degree values ​​1, 2, and 3 of the connection data blocks, the largest in-degree value 3 can be selected, thereby determining 8 as the target data block. The target data block 8 is stored in a cache associated with the database.

[0078] Optionally, the target data block may be selected from the connected data blocks by paying attention to the in-degree value, or paying attention to the number of times the data block appears before and after, the time sequence of appearance, etc., but it is not limited to this, and the embodiments of the present invention do not give examples one by one.

[0079] Optionally, determining the target data block according to the in-degree values ​​of the connected data blocks includes: obtaining a target query statement when the in-degree values ​​of each of the multiple connected data blocks satisfy a preset in-degree value condition; parsing the target query statement to obtain a target query plan tree; executing the query plan of the target query plan tree to obtain an intermediate execution data block; and determining a connected data block among the multiple connected data blocks that matches the intermediate execution data block as the target data block.

[0080] Optionally, when the in-degree values ​​of the multiple connection data blocks all meet the preset in-degree value conditions, the target query statement can be obtained. By parsing the target query statement, a target query plan tree can be obtained; by executing the query plan of the target query plan tree, an intermediate execution data block can be obtained; and the connection data block that matches the intermediate execution data block among the multiple connection data blocks is determined as the target data block. For example, the cache has: 1, 2, 3, 4, 5. For the next pre-stored buffer block, it can be seen in the directed topology graph that the connection data blocks 6 and 7 have an execution logic relationship with the cache data block 5, and the in-degree values ​​of the connection data blocks 6 and 7 are the same. The target query statement subselect1 can be obtained, the target query plan tree can be obtained, the query plan of the target query plan tree can be executed, and the intermediate execution data blocks (1, 2, 3, 4, 5, 6) can be obtained. From the intermediate target data block of the target query statement subselect1, it is determined that the next data block to be placed in the cache is 6.

[0081] Figure 4A-4B The diagram schematically shows a data block relationship network diagram according to an embodiment of the present invention.

[0082] like Figure 4AAs shown, subselect1, subselect2, subselect3, and subselect4 can represent different query statements, and datacell1, datacell3, datacell5, datacell11, datacell19…datacellX, etc. represent different data blocks. It can be seen that the data blocks with execution logic relationships corresponding to subselect1 are datacell1—> datacell3—> datacell5…datacellX. It can be seen that the data blocks with execution logic relationships corresponding to subselect2 are datacell1—> datacell3—> datacell11…datacellX. It can be seen that the data blocks with execution logic relationships corresponding to subselect3 are datacell1—> datacell5—>datacell11…datacellX. It can be seen that the data blocks with execution logic relationships corresponding to subselect4 are datacell3—> datacell5—> datacell19…datacellX. A directed topology graph can be constructed based on data blocks and the execution logic relationships between data blocks, such as Figure 4B As shown. Figure 4B The connection relationship and sequence between the data blocks are shown.

[0083] Optionally, suppose there are currently two data blocks in the data cache, datacell1 and datacell3. If prefetched in disk order, the next data block to enter the cache should be datacell2 or datacell4. In this case, there is no required data block in the cache, and the cache hit rate is 0. However, we can see from the above query statements (i.e., subselect1, subselect2, subselect3, and subselect4) and the relationship between the data blocks that the query statements do not involve datacell2 and datacell4. The data blocks related to datacell1 are datacell3 and datacell5, and the data blocks related to datacell3 are datacell11 and datacell5. Therefore, the set of data blocks related to the two data blocks datacell1 and datacell3 in the cache is<datacell3, datacell5, datacell11> . datacell3 is already in the data cache, so pre-fetching is excluded; the in-degree value of datacell5 is 3; the in-degree value of datacell11 is 1 (only datacell3->datacell11 is included, but not datacell5->datacell11); the weight of the relation datacell5 that is not in the cache is greater than that of datacell11, so datacell5 is taken as the next pre-fetched data block (by pre-fetching data blocks according to the optional method, the cache hit rate can reach 50%).

[0084] According to the data processing method proposed by the present invention, the common subtree is used as the basic query unit, and the data block is used as the node of the directed topology graph to build the network relationship between the data blocks. With the repeated query of the database query statement and the iterative update of the data block relationship graph, the relationship between the data blocks is more accurate and the hit rate will gradually increase.

[0085] Based on the above data processing method, the present invention also provides a data processing device. Figure 5 The device is described in detail.

[0086] Figure 5 The structure block diagram of the data processing device according to the embodiment of the present invention is schematically shown.

[0087] like Figure 5 As shown, the data processing device 500 of this embodiment includes a parsing module 510 , a data block obtaining module 520 , a directed topology graph establishing module 530 , a query module 540 and a storage module 550 .

[0088] The parsing module 510 is used to parse the query statement and obtain a query plan tree.

[0089] The data block obtaining module 520 is used to execute the query plan of the query plan tree to obtain multiple data blocks.

[0090] The directed topology graph establishing module 530 is used to establish a directed topology graph according to the data blocks and the execution logic relationships between the data blocks.

[0091] A query module 540 is used to query a target data block having an execution logic relationship with a cache data block from the directed topology graph, wherein the cache data block is stored in a cache associated with a database.

[0092] The storage module 550 is used to store the target data block in the cache associated with the database.

[0093] Optionally, by parsing the query statement, a query plan tree is obtained, and the query plan of the query plan tree is executed to obtain multiple data blocks. A directed topology graph is established based on the data blocks and the connection relationship between the data blocks. Finally, the next data block to be pre-fetched is searched from the directed topology graph based on the cache data blocks existing in the cache. In the pre-fetching process, a directed topology graph is established by considering the relationship between the query statement and the data block. The directed topology graph can be used to predict the data blocks that the query statement will read in the future, thereby strengthening the data cache pre-fetching in the cache management, reducing the cache miss rate, and improving the cache hit rate.

[0094] Optionally, the data block obtaining module 520 includes: a determination submodule and an execution submodule.

[0095] A sub-module is used to determine a common sub-tree from multiple query plan trees;

[0096] The execution submodule is used to execute the query plan of the common subtree to obtain multiple data blocks.

[0097] Optionally, the determination submodule includes: a matching unit and a common subtree determination unit.

[0098] A matching unit, used for matching a reference query subtree in the reference query plan tree with a benchmark query subtree in the benchmark query plan tree according to a matching rule to obtain a matching result, wherein the plurality of query plan trees include the reference query plan tree and the benchmark query plan tree;

[0099] The common subtree determination unit is used to determine the reference query subtree or the benchmark query subtree as a common subtree when the matching result representation matches.

[0100] Optionally, the matching rule represents that: the connection conditions of the base query subtree and the reference query subtree are the same; and the query subnode of the base query subtree and the query subnode of the reference query subtree are the same.

[0101] Optionally, the query module 540 includes: a submodule for obtaining a connected data block, a submodule for determining an in-degree value, and a submodule for determining a target data block.

[0102] The connection data block obtaining submodule is used to search from the directed topology graph according to the cache data block to obtain multiple connection data blocks, and the cache data block and the connection data block have an execution logic relationship.

[0103] The in-degree determination submodule is used to determine the in-degree values ​​of the connected data blocks in the directed topology graph according to the execution logic relationship between the data blocks.

[0104] The target data block determination submodule is used to determine the target data block according to the in-degree value of the connection data block.

[0105] Optionally, the target data block determination submodule includes: a target acquisition unit, a target parsing unit, an intermediate execution unit and a target data block determination unit.

[0106] The target acquisition unit is used to acquire the target query statement when the in-degree values ​​of the multiple connected data blocks all meet the preset in-degree value conditions.

[0107] The target parsing unit is used to parse the target query statement and obtain the target query plan tree.

[0108] The intermediate execution unit is used to execute the query plan of the target query plan tree and obtain the intermediate execution data block.

[0109] The target data block determining unit is used to determine the connection data block that matches the intermediate execution data block among the multiple connection data blocks as the target data block.

[0110] Optionally, any multiple modules among the parsing module 510, the data block obtaining module 520, the directed topology map establishing module 530, the query module 540 and the storage module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the parsing module 510, the data block obtaining module 520, the directed topology map establishing module 530, the query module 540 and the storage module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Or at least one of the parsing module 510, the data block obtaining module 520, the directed topology graph building module 530, the query module 540 and the storage module 550 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0111] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is schematically shown.

[0112] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0113] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0114] Optionally, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0115] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0116] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, optionally, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0117] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present invention.

[0118] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0119] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0120] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. Optionally, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0121] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0122] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0124] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Parse the query statement and obtain the query plan tree; Execute the query plan of the query plan tree to obtain multiple data blocks; Establishing a directed topology graph according to the data blocks and the execution logic relationships between the data blocks; Querying a target data block having an execution logic relationship with a cache data block from the directed topological graph, wherein the cache data block is stored in a cache associated with a database; storing the target data block in the cache associated with the database, The step of searching the directed topology graph for a target data block having an execution logic relationship with the cache data block includes: Searching the directed topology graph according to the cache data block to obtain a plurality of connection data blocks, wherein the cache data block and the connection data block have an execution logic relationship; Determining the in-degree values ​​of the connected data blocks in the directed topological graph according to the execution logic relationship between the data blocks; Determining the target data block according to the in-degree value of the connection data block; The step of determining the target data block according to the in-degree value of the connection data block includes: When the in-degree values ​​of the plurality of connected data blocks all meet the preset in-degree value condition, obtaining a target query statement; Parsing the target query statement to obtain a target query plan tree; Execute the query plan of the target query plan tree to obtain an intermediate execution data block; A connection data block among the plurality of connection data blocks that matches the intermediate execution data block is determined as the target data block.

2. The method according to claim 1, characterized in that The executing the query plan of the query plan tree to obtain a plurality of data blocks comprises: Determine a common subtree from a plurality of the query plan trees; The query plan of the common subtree is executed to obtain a plurality of the data blocks.

3. The method according to claim 2, characterized in that Determining a common subtree from the plurality of query plan trees comprises: According to the matching rule, matching the reference query subtree in the reference query plan tree with the benchmark query subtree in the benchmark query plan tree to obtain a matching result, wherein the plurality of query plan trees include the reference query plan tree and the benchmark query plan tree; In the case where the matching result representation is matched, the reference query subtree or the benchmark query subtree is determined as the common subtree.

4. The method according to claim 3, characterized in that The matching rule characterizes: The benchmark query subtree and the reference query subtree have the same connection conditions; and The query sub-nodes of the base query sub-tree are the same as the query sub-nodes of the reference query sub-tree.

5. A data processing device, characterized in that: The device comprises: The parsing module is used to parse the query statement and obtain the query plan tree; A data block obtaining module, used for executing the query plan of the query plan tree to obtain multiple data blocks; A directed topology graph establishing module, used for establishing a directed topology graph according to the data blocks and the execution logic relationships between the data blocks; a query module, configured to query the directed topological graph for a target data block having an execution logic relationship with a cache data block, wherein the cache data block is stored in a cache associated with a database; and A storage module, used for storing the target data block in the cache associated with the database; Wherein, the query module includes: A connection data block obtaining submodule, used for searching from the directed topology graph according to the cache data block to obtain a plurality of connection data blocks, wherein the cache data block and the connection data block have an execution logic relationship; An in-degree determination submodule, used to determine the in-degree values ​​of the connection data blocks in the directed topology graph according to the execution logic relationship between the data blocks; A target data block determination submodule, used to determine the target data block according to the in-degree value of the connection data block; Wherein, the target data block determination submodule includes: A target acquisition unit, configured to acquire a target query statement when the in-degree values ​​of the plurality of connected data blocks all meet a preset in-degree value condition; A target parsing unit, used for parsing a target query statement to obtain a target query plan tree; An intermediate execution unit, used for executing the query plan of the target query plan tree and obtaining an intermediate execution data block; The target data block determining unit is used to determine the connection data block that matches the intermediate execution data block among the multiple connection data blocks as the target data block.

6. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images