Logic data block-based data query method and apparatus

Abstract

The present application provides a logic data block-based data query method and apparatus, used for realizing query acceleration of data query in a database by means of a logic data block. The method relates to a client, a server, and an execution end. The method comprises: the execution end receives a data query request sent by the client, and by means of an optimizer and on the basis of the data query request, generates an AcScan operator; on the basis of the AcScan operator, the execution end scans an acceleration data block in the server, and determines by means of the scanning whether the acceleration data block is valid or not; and when it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block on the basis of a function G and a received mapping relationship list sent by the server, and performs calculation on the basis of the target acceleration data block until query is completed. In this way, the result correctness of concurrent service query is ensured, the acceleration effect is improved, and rapid refresh of a result set or an intermediate result set is realized.

Description

A data query method and device based on logical data blocks Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data query method and device based on logical data blocks. Background Art

[0002] With the advancement of digital transformation across industries, enterprises are facing increasing demands for data processing and real-time data availability. The need for real-time analysis of massive amounts of data is becoming increasingly prominent. In real-time analysis of massive amounts of data, certain complex analytical statements often take a particularly long time to process. Accelerating queries for these specific statements is a key requirement in analytics. Traditional processing methods typically use distributed computing to divide and conquer, but this consumes a significant amount of computing resources and is prohibitively expensive. Therefore, improving the efficiency of complex queries within a specific unit of computing resources and adapting to concurrent data modifications in real-time analysis have become key competitive advantages.

[0003] In massive data analysis scenarios, accelerating complex queries within a specific unit of computing resources is a key issue. A common approach is to trade space for time by caching query result sets or intermediate result sets. Complex queries are accelerated by accessing these cached result sets or intermediate result sets. This requires addressing the issue of outdated result sets or intermediate result sets due to real-time data modifications, which can lead to incorrect results.

[0004] The existing technical solution is implemented through Oracle materialized views. Oracle materialized views pre-calculate result sets using SQL statements and store them in materialized view objects. When a business statement requires this result set, the optimizer rewrites the statement to obtain the result set from the materialized view, eliminating the need to query data from the original table and perform further calculations. This accelerates the process. Materialized views are commonly used in historical database analysis scenarios, but their update capabilities are significantly limited, making them significantly lacking in real-time analysis scenarios involving update operations. Ensuring the accuracy and acceleration of concurrent business query results, as well as rapidly refreshing result sets or intermediate result sets, are technical issues that require a solution.

[0005] Summary of the Invention

[0006] In view of the above problems, the present application is proposed to provide a data query method based on logical data blocks and a device thereof that overcomes the above problems or at least partially solves the above problems, including:

[0007] A data query method based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The method involves a client, a server, and an execution end, and includes: the execution end receiving a query data request sent by the client, and generating an AcScan operator through an optimizer based on the query data request;

[0008] The execution end scans the acceleration data block in the server according to the AcScan operator, and determines whether the acceleration data block is valid through the scanning;

[0009] When it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on the function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

[0010] Furthermore, the step of generating an AcScan operator through an optimizer according to the query data request includes:

[0011] The execution end determines whether to use the acceleration data block in the server according to the cost evaluation of the query data request by the optimizer;

[0012] When it is determined to use the acceleration data block, the execution end determines an operator tree corresponding to the acceleration data block;

[0013] The execution end generates the AcScan operator for scanning the acceleration data block according to the operator tree.

[0014] Furthermore, the step of constructing a target acceleration data block according to the function G and the mapping relationship list includes:

[0015] The execution end determines the logical data block corresponding to the acceleration data block in the server through a mapping relationship list according to the function G;

[0016] The execution end determines a Scan operator corresponding to the logical data block according to the logical data block;

[0017] The execution end scans and determines the data in the logical data block through the Scan operator;

[0018] The execution end converts the format of the data to generate the target acceleration data block.

[0019] Furthermore, the method further comprises:

[0020] When the query is completed, the execution end performs calculation based on the target acceleration data block or the acceleration data block to generate a calculation result;

[0021] The execution end generates a query result according to the calculation result, and sends the query result to the client.

[0022] The present application also discloses a data query method based on logical data blocks, which is used to accelerate data query in a database through the logical data blocks. The logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The method involves a client, a server, and an execution end, and includes:

[0023] The server receives the creation command sent by the database and constructs a data model conversion rule according to the creation command;

[0024] The server sends the data model conversion rule to the database, constructs the mapping relationship list based on the logical data block, and sends the mapping relationship list to the execution end;

[0025] The server determines, according to the mapping relationship list, whether the logical data block has the acceleration data block corresponding to the logical data block;

[0026] When the logical data block does not have a corresponding acceleration data block, the logical data block is marked as a first data block to be generated, and the server generates a first newly constructed acceleration block corresponding to the first data block to be generated according to the data model conversion rule;

[0027] The server marks the first newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the first newly constructed acceleration block.

[0028] Furthermore, the method further comprises:

[0029] After the data in the logical data block is modified, the server marks the acceleration data block corresponding to the logical data block as invalid, and marks all mapping relationships related to the acceleration data block in the mapping relationship list as invalid.

[0030] The server determines a logical data block corresponding to the acceleration data block by using the acceleration data block marked as the failure state;

[0031] The server marks the logical data block as a second data block to be generated, and generates a second newly constructed acceleration block corresponding to the second data block to be generated according to the data model conversion rule;

[0032] The server marks the second newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the second newly constructed acceleration block.

[0033] The present application also discloses a data query method based on logical data blocks, which is used to accelerate data query in a database through the logical data blocks. The logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The method involves a client, a server, and an execution end, and includes:

[0034] The client generates a query data request according to user needs and sends the query data request to the execution end;

[0035] The client receives the query result sent by the execution end and feeds the query result back to the corresponding user.

[0036] The present application also discloses a data query device based on logical data blocks, which is used to accelerate data queries in a database using the logical data blocks. The logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The device involves a client, a server, and an execution end, and includes:

[0037] A first generating module is configured to receive, at the execution end, a query data request sent by the client, and generate an AcScan operator through an optimizer according to the query data request;

[0038] A first determination module is configured to enable the execution end to scan the acceleration data block in the server according to the AcScan operator, and determine whether the acceleration data block is valid through the scanning;

[0039] A first computing module is configured to, when determining that the acceleration data block is invalid, cause the execution end to construct a target acceleration data block based on a function G and a mapping relationship list received from the server, and perform calculations based on the target acceleration data block until the query is completed; wherein, the function G is configured to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

[0040] The present application also discloses a data query device based on logical data blocks, which is used to accelerate data queries in a database using the logical data blocks. The logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The device involves a client, a server, and an execution end, and includes:

[0041] A first building module is configured for the server to receive a creation command sent by the database and build a data model conversion rule according to the creation command;

[0042] A second building module is used for the server to send the data model conversion rule to the database, build the mapping relationship list based on the logical data block, and send the mapping relationship list to the execution end;

[0043] A second determining module is configured for the server to determine, based on the mapping relationship list, whether the logical data block has the acceleration data block corresponding to the logical data block;

[0044] A second generating module is configured to, when the corresponding acceleration data block does not exist for the logical data block, mark the logical data block as a first data block to be generated, and generate, by the server, a first newly constructed acceleration block corresponding to the first data block to be generated according to the data model conversion rule;

[0045] The first marking module is configured to mark, by the server, the first newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the first newly constructed acceleration block.

[0046] The present application also discloses a data query device based on logical data blocks, which is used to accelerate data queries in a database using the logical data blocks. The logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The device involves a client, a server, and an execution end, and includes:

[0047] A third generating module is used for the client to generate a query data request according to user needs and send the query data request to the execution end;

[0048] The first feedback module is used for the client to receive the query result sent by the execution end and feed back the query result to the corresponding user.

[0049] This application has the following advantages:

[0050] In an embodiment of the present application, relative to the prior art problem of "how to ensure the correctness of the results of concurrent business queries and the acceleration effect, and how to quickly refresh the result set or intermediate result set are problems that need to be solved technically", the present application provides a solution of "logical data blocks and acceleration data blocks", specifically: "the execution end receives the query data request sent by the client, and generates an AcScan operator through the optimizer based on the query data request; the execution end scans the acceleration data block in the server based on the AcScan operator, and determines whether the acceleration data block is valid through the scan; when it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block." Through "logical data blocks and acceleration data blocks", the problem of "how to ensure the correctness of the results of concurrent business queries and the acceleration effect, and how to quickly refresh the result set or intermediate result set are problems that need to be solved technically" is solved, and "business queries can adaptively partially access acceleration data blocks and partially access logical data blocks, which can maintain local acceleration capabilities, and the query can return correct results to meet transaction consistency requirements. At the same time, by being able to automatically build temporarily invalid acceleration data blocks in the background, after the construction is completed, the acceleration data block can be used to accelerate business queries. Each acceleration data block can be built independently, and once an acceleration data block is built, it can be used for acceleration without waiting for all acceleration data blocks to be built. It ensures the correctness of the results of concurrent business queries, improves the acceleration effect, and achieves rapid refresh of result sets or intermediate result sets." BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] FIG1 is a flowchart of a data query method based on logical data blocks provided by an embodiment of the present application;

[0053] FIG2 is a flowchart of a data query method based on logical data blocks provided by an embodiment of the present application;

[0054] FIG3 is a flowchart of a data query method based on logical data blocks provided by an embodiment of the present application;

[0055] FIG4 is a structural block diagram of a data query device based on logical data blocks provided by an embodiment of the present application;

[0056] FIG5 is a structural block diagram of a data query device based on logical data blocks provided by an embodiment of the present application;

[0057] FIG6 is a structural block diagram of a data query device based on logical data blocks provided by an embodiment of the present application;

[0058] FIG7 is a schematic structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.

[0060] The inventors found through analysis of the prior art that:

[0061] Existing technical solution 1:

[0062] Implemented through Oracle materialized views;

[0063] Oracle materialized views use SQL statements to pre-calculate result sets and store them in materialized view objects. When a business statement requires this result set, the optimizer can rewrite the statement to obtain the result set from the materialized view without having to query data from the original table and then perform calculations. This can accelerate the process. Materialized views are commonly used in historical database analysis scenarios, but their update capabilities are quite limited, making them a significant disadvantage for real-time analysis scenarios with update operations. Disadvantages of this solution:

[0064] There are several ways to refresh a materialized view after the original table data is modified, but all of them have some disadvantages.

[0065] 1. Full refresh method: After the original table data is modified, the materialized view is fully refreshed, which is very costly.

[0066] 2. Changelog-based incremental refresh: Row-level change logs are recorded and merged into the existing materialized view during refresh. Storing change logs requires additional overhead; additionally, when the number of changes is large, the cost of incremental refresh is also high. When performing operations such as partition maintenance on partitioned tables, this refresh method cannot be used because row-level change logs cannot be recorded, requiring a full refresh.

[0067] 3. Partition change tracking (PCT) based on partitioned tables: This method identifies modified partitions based on the partition key and other factors. Materialized views are refreshed only for the affected partitions, while materialized views for unaffected partitions continue to provide accelerated services. This method provides faster refresh capabilities, but PCT has the following limitations, making its use less perceptible to the business layer and less widely applicable.

[0068] a) The original table must be a partitioned table.

[0069] b) The partition type must be range or list, or a combination of range / list types at the first level.

[0070] c) A partition key can contain only one field.

[0071] d) The materialized view query columns must include the partition key field, partition tag, or ROWID.

[0072] e) If the materialized view contains a group by clause, the partition key field / partition marker / ROWID must also be in the group by column.

[0073] Therefore, according to the existing technical solutions, there are still problems that need to be solved technically, such as how to ensure the correctness and acceleration effect of the results of concurrent business queries, and how to quickly refresh the result set or intermediate result set.

[0074] 1 , a flowchart of a data query method based on logical data blocks according to an embodiment of the present application is shown;

[0075] A data query method based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The method involves a client, a server, and an execution end, and includes:

[0076] S110, the execution end receives the query data request sent by the client, and generates an AcScan operator through the optimizer according to the query data request;

[0077] S120: The execution end scans the acceleration data block in the server according to the AcScan operator, and determines whether the acceleration data block is valid through the scanning;

[0078] S130. When it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on the function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

[0079] In an embodiment of the present application, relative to the prior art problem of "how to ensure the correctness of the results of concurrent business queries and the acceleration effect, and how to quickly refresh the result set or intermediate result set are problems that need to be solved technically", the present application provides a solution of "logical data blocks and acceleration data blocks", specifically: "the execution end receives the query data request sent by the client, and generates an AcScan operator through the optimizer based on the query data request; the execution end scans the acceleration data block in the server based on the AcScan operator, and determines whether the acceleration data block is valid through the scan; when it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block." Through "logical data blocks and acceleration data blocks", the problem of "how to ensure the correctness of the results of concurrent business queries and the acceleration effect, and how to quickly refresh the result set or intermediate result set are problems that need to be solved technically" is solved, and "business queries can adaptively partially access acceleration data blocks and partially access logical data blocks, which can maintain local acceleration capabilities, and the query can return correct results to meet transaction consistency requirements. At the same time, by being able to automatically build temporarily invalid acceleration data blocks in the background, after the construction is completed, the acceleration data block can be used to accelerate business queries. Each acceleration data block can be built independently, and once an acceleration data block is built, it can be used for acceleration without waiting for all acceleration data blocks to be built. It ensures the correctness of the results of concurrent business queries, improves the acceleration effect, and achieves rapid refresh of result sets or intermediate result sets."

[0080] Next, a data query method based on logical data blocks in this exemplary embodiment will be further described.

[0081] In one embodiment of the present invention, logical data blocks: source data is divided into fine-grained data blocks for storage and management. Each data block is called a logical data block Slice. This division is completed by the database system and is transparent to the business layer.

[0082] In one embodiment of the present invention, the accelerated data block: Each logical data block is converted into a data model according to the defined conversion rules to generate an accelerated data block AcSlice. Services accessing the accelerated data block achieve a speed-up effect, and the results are consistent with those of accessing the logical data block.

[0083] In one embodiment of the present invention, Scan operator: a method for scanning data in a logical data block.

[0084] In one embodiment of the present invention, the AcSacn operator is a method for scanning data in an accelerated data block.

[0085] In one embodiment of the present invention, the source data within the database is divided into fine-grained logical data blocks, called Slices. By defining a data model conversion rule function F(D), each logical data block generates an accelerated data block, called an AcSlice. The mapping between logical data blocks and accelerated data blocks can be one-to-one or many-to-one. The conversion rules can include conditions to perform model conversion on only part of the data in a logical data block, or they can convert only some logical data blocks based on resource allocation.

[0086] In one embodiment of the present invention, when querying data, the operator of the logical data block Slice of the scan source data is defined as Scan, the operator of the accelerated data block AcSlice after scanning conversion is defined as AcScan, and the function G is defined so that G(AcScan)=Scan, and Scan(Slice) and AcScan(AcSlice) are logically equivalent in data.

[0087] In one embodiment of the present invention, regardless of whether the acceleration data block construction has been completed, an adaptive query acceleration process can be initiated when a query is generated. This process can adaptively scan the acceleration data blocks and logical data blocks, even if the acceleration data blocks are fully or partially valid, to obtain correct query results while maintaining the maximum possible acceleration effect.

[0088] In one embodiment of the present invention, when it is determined that the acceleration data block is valid, the execution end performs calculations based on the acceleration data block until the query is completed; when the acceleration data block is valid, the acceleration data block can be directly used for calculations.

[0089] In one embodiment of the present invention, the step of generating an AcScan operator through an optimizer according to the query data request includes:

[0090] The execution end determines whether to use the acceleration data block in the server according to the cost evaluation of the query data request by the optimizer;

[0091] When it is determined to use the acceleration data block, the execution end determines an operator tree corresponding to the acceleration data block;

[0092] The execution end generates the AcScan operator for scanning the acceleration data block according to the operator tree.

[0093] In a specific embodiment, when querying data, through the optimizer cost evaluation, if the operator tree of the accelerated data block is selected, an AcScan operator is generated to scan the accelerated data block; the execution end executes the AcScan operator to scan the accelerated data block AcSlice.

[0094] In one embodiment of the present invention, the step of constructing a target acceleration data block according to the function G and the mapping relationship list includes:

[0095] The execution end determines the logical data block corresponding to the acceleration data block in the server through a mapping relationship list according to the function G;

[0096] The execution end determines a Scan operator corresponding to the logical data block according to the logical data block;

[0097] The execution end scans and determines the data in the logical data block through the Scan operator;

[0098] The execution end converts the format of the data to generate the target acceleration data block.

[0099] In a specific embodiment, when query acceleration is executed, for the converted data, AcScan is called to scan the acceleration data block AcSlice. For the case of unconverted or temporary failure of AcSlice, function G switches to scanning the corresponding logical data block Slice, thereby achieving consistency in the final result and realizing query acceleration for the converted data.

[0100] In one specific implementation, when an accelerated data block fails, the call to function G is converted to calling the Scan operator to scan the logical data block Slice of the source data, replacing the AcScan operator. This maintains the data logical equivalence between Scan (Slice) and AcScan (AcSlice). This conversion is transparent to the optimizer. Once the background task completes the reconstruction of the accelerated data block, it can return to the AcScan operator. This enables adaptive switching queries between logical data blocks and accelerated data blocks without causing business queries to fail, while maximizing the acceleration effect. The data obtained by scanning the logical data block using the Scan operator is converted to the accelerated data block format to accommodate subsequent operators in the operator tree. The accelerated data block is then returned for computation until the query is completed.

[0101] In one embodiment of the present invention, the method further includes:

[0102] When the query is completed, the execution end performs calculation based on the target acceleration data block or the acceleration data block to generate a calculation result;

[0103] The execution end generates a query result according to the calculation result, and sends the query result to the client.

[0104] 2 , there is shown a flowchart of a data query method based on logical data blocks according to an embodiment of the present application;

[0105] A data query method based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The method involves a client, a server, and an execution end, and includes:

[0106] S210, the server receives a creation command sent by the database and constructs a data model conversion rule according to the creation command;

[0107] S220: The server sends the data model conversion rule to the database, constructs the mapping relationship list based on the logical data block, and sends the mapping relationship list to the execution end;

[0108] S230: The server determines, according to the mapping relationship list, whether the logical data block has the acceleration data block corresponding to the logical data block;

[0109] S240: When the corresponding acceleration data block does not exist for the logical data block, marking the logical data block as a first data block to be generated, and generating, by the server according to the data model conversion rule, a first newly constructed acceleration block corresponding to the first data block to be generated;

[0110] S250: The server marks the first newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the first newly constructed acceleration block.

[0111] It should be noted that the server divides the source data into multiple fine-grained logical data blocks for storage and management. This division is transparent to the business layer. Conversion rules are defined to perform data model conversion and generate acceleration data blocks. There is a one-to-one or many-to-one mapping relationship between logical data blocks and acceleration data blocks.

[0112] In a specific embodiment, data model conversion rules are defined based on information such as access fields and conditions of a database in a complex query, and the data model conversion rules are created using a command provided by the database system, namely a create command. The created data model conversion rules are recorded within the database. At the same time, a mapping relationship list between logical data blocks and acceleration data blocks is created. The mapping relationship list includes the mapping relationship between each logical data block and the acceleration data block, as well as the effectiveness status of the acceleration data block. From this, it is possible to see from the mapping relationship list which acceleration data blocks have expired and which logical data blocks need to be used to construct the acceleration data block. A background task is notified to asynchronously perform data model conversion and generate acceleration data blocks. After receiving the notification, the background thread starts executing the task and reads the data model conversion rules. Based on the mapping relationship and effectiveness status between the logical data block and the acceleration data block, the ineffective acceleration data block and the corresponding logical data block are identified. One or more logical data blocks are scanned and an acceleration data block is generated according to the conversion rule. Based on the mapping relationship between the logical data block and the acceleration data block, the generated acceleration data block is stored as an independent acceleration data block or merged into an existing acceleration data block. Each time the construction of an acceleration data block is completed, the acceleration data block is set to a valid state in the mapping relationship.

[0113] In one embodiment of the present invention, the method further includes:

[0114] After the data in the logical data block is modified, the server marks the acceleration data block corresponding to the logical data block as invalid, and marks all mapping relationships related to the acceleration data block in the mapping relationship list as invalid.

[0115] The server determines a logical data block corresponding to the acceleration data block by using the acceleration data block marked as the failure state;

[0116] The server marks the logical data block as a second data block to be generated, and generates a second newly constructed acceleration block corresponding to the second data block to be generated according to the data model conversion rule;

[0117] The server marks the second newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the second newly constructed acceleration block.

[0118] In a specific embodiment, when data is modified, the corresponding acceleration data block is set to an invalid state based on the modified logical data block. At the same time, all mapping relationships related to this acceleration data block are found in the mapping relationship list and set to an invalid state. In this way, it is possible to know which logical data blocks need to be used to rebuild the acceleration data block; the background task is notified to asynchronously perform data model conversion, identify the invalid acceleration data block, and rebuild it.

[0119] 3 , there is shown a flowchart of a data query method based on logical data blocks provided by an embodiment of the present application;

[0120] A data query method based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The method involves a client, a server, and an execution end, and includes:

[0121] S310: The client generates a query data request according to user needs and sends the query data request to the execution end;

[0122] S320: The client receives the query result sent by the execution end, and feeds back the query result to the corresponding user.

[0123] In a specific embodiment, the client generates a query data request according to user needs, sends the query data request to the execution end, receives the query result sent by the execution end, and feeds back the query result to the corresponding user.

[0124] Example 1

[0125] Here, we analyze the capacity, actual number of passengers, and delay summary information of each operator and route for domestic flights in Country A. The query statement is as follows:

[0126] This involves two tables: the segment table and the on_time table. The segment table contains information such as flight routes, loads, and passenger numbers; the on_time table contains information such as flight delays. When creating tables in this system, table data is divided into fine-grained logical data blocks for storage, with each logical data block containing 2 million records.

[0127] Based on the characteristics of business statements, we create the following data model conversion rules for the segment table and on_time table respectively.

[0128] Create a transformation rule function F1(D) on the segment table:

[0129] F1(D)=(ORIGIN_CITY_NAME,UNIQUE_CARRIER_NAME,AIRLINE_ID, DEST_COUNTRY,ORIGIN_COUNTRY,COUNT,SUM(PASSENGERS),SUM(SEATS))

[0130] Create a conversion rule function F2(D) on the on_time table:

[0131] F2(D)=(OriginCityName,DOT_ID_Reporting_Airline,COUNT,SUM(DepDelayMinutes),SUM(ArrDelayMinutes))

[0132] The segment table generates corresponding acceleration data blocks from each logical data block based on the conversion rule function F1(D), and the on_time table generates corresponding acceleration data blocks based on the conversion rule function F2(D). The number of records in the acceleration data blocks is much smaller than the number of records in the logical data blocks, and the acceleration data blocks contain the COUNT and SUM aggregate values ​​calculated in the logical data blocks.

[0133] In the query plan tree generated by the above query statement, the scan operators for the segment and on_time tables use the AcScan operator, which scans the acceleration data block. By scanning the segment table's acceleration data block, the AcScan operator obtains the ORIGIN_CITY_NAME, UNIQUE_CARRIER_NAME, AIRLINE_ID, DEST_COUNTRY, and ORIGIN_COUNTRY columns, as well as the SUM(PASSENGERS) and SUM(SEATS) values ​​required for the calculation. By scanning the on_time table's acceleration data block, the AcScan operator obtains the OriginCityName and DOT_ID_Reporting_Airline columns, as well as the SUM(DepDelayMinutes) and SUM(ArrDelayMinutes) values ​​required for the calculation. The data obtained from each acceleration data block is then merged and superimposed to obtain the final calculation result.

[0134] Taking the segment table as an example, when a logical data block is modified, the corresponding acceleration data block is marked as invalid. During concurrent queries, if the acceleration data block is found to be invalid, function G automatically switches to calling the Scan operator to scan the logical data block. The scan results extract the ORIGIN_CITY_NAME, UNIQUE_CARRIER_NAME, AIRLINE_ID, DEST_COUNTRY, and ORIGIN_COUNTRY columns, and add COUNT, SUM(PASSENGERS), and SUM(SEATS) values ​​to each scanned record. This is assembled into the AcScan operator query result format and then passed to the upper-level operator for calculation. Here, for each record, COUNT is 1, SUM(PASSENGERS) is the original value of PASSENGERS, and SUM(SEATS) is the original value of SEATS. A similar process occurs when data is modified in the on_time table.

[0135] For the acceleration data block in the failed state, the corresponding logical data block is found through the mapping relationship list, and the background automatically rebuilds the acceleration data block.

[0136] Observe that the query statement for the segment table contains the constant condition ORIGIN_COUNTRY = 'US'. If this condition is frequently used in complex analysis statements, we can create a transformation rule function F1`(D) with the condition on the segment table, that is, to build an accelerated data block only for data that meets the condition:

[0137] F1`(D)=(ORIGIN_CITY_NAME,UNIQUE_CARRIER_NAME,AIRLINE_ID,DEST_COUNTRY,ORIGIN_COUNTRY,COUNT,SUM(PASSENGERS),SUM(SEATS))@ORIGIN_COUNTRY='US'

[0138] This conversion rule function reduces the number of records in the accelerated data block, resulting in a more pronounced acceleration effect. However, this also limits its applicability. Query statements that don't meet this condition cannot use this accelerated data block. Therefore, it's generally recommended to only add conditions commonly used in complex analysis statements to the conversion rule function.

[0139] Through configuration or other means, N logical data blocks are mapped to one acceleration data block, forming an N-to-1 mapping relationship. This reduces the number of acceleration data blocks and improves the acceleration effect. However, the corresponding trade-off is that when data is modified, the N logical data blocks corresponding to the invalid acceleration data block will temporarily lose their acceleration capabilities and need to be rebuilt from the N logical data blocks. Therefore, it is generally recommended to use the N-to-1 approach to generate acceleration data blocks when data modifications are rare.

[0140] Technical effects of the present invention:

[0141] 1) This solution provides a data query method based on logical data blocks, which divides source data into fine-grained logical data blocks. The logical data block division is transparent to the application layer. Based on information such as database access fields and conditions in complex queries, conversion rules are defined to perform data model conversion and generate acceleration data blocks. Logical data blocks and acceleration data blocks have a one-to-one or many-to-one mapping relationship, enabling independent acceleration of each logical data block. Conversion rules can include conditions to perform model conversion on only a portion of the data in a logical data block, or they can be applied to only a portion of the logical data blocks based on resource allocation. When data in a portion of a logical data block is modified, only the mapped acceleration data blocks are temporarily disabled, without incurring transaction lock waits. Business queries can adaptively access only acceleration data blocks and only logical data blocks, maintaining local acceleration capabilities. Queries can also return correct results and meet transaction consistency requirements. Simultaneously, the temporarily disabled acceleration data blocks are automatically constructed in the background. Once constructed, these acceleration data blocks can be used to accelerate business queries. Each acceleration data block can be constructed independently; once constructed, they can be used for acceleration without waiting for all acceleration data blocks to be completed.

[0142] 2) Different acceleration data blocks in this solution can use different data structures as needed.

[0143] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0144] 4 , there is shown a block diagram of a data query device based on logical data blocks according to an embodiment of the present application;

[0145] A data query device based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The device involves a client, a server, and an execution end, and includes:

[0146] A first generating module 410 is configured to receive, at the execution end, a query data request sent by the client, and generate an AcScan operator through an optimizer according to the query data request;

[0147] A first determining module 420 is configured to enable the execution end to scan the acceleration data block in the server according to the AcScan operator, and determine whether the acceleration data block is valid through the scanning;

[0148] The first calculation module 430 is used to, when it is determined that the acceleration data block is invalid, construct a target acceleration data block based on the function G and the mapping relationship list sent by the server, and perform calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

[0149] In one embodiment of the present invention, the first generating module 410 includes:

[0150] A first determining submodule, configured for the execution end to determine whether to use the acceleration data block in the server based on a cost evaluation of the query data request by the optimizer;

[0151] a second determining submodule, configured to, when determining to use the acceleration data block, cause the execution end to determine an operator tree corresponding to the acceleration data block;

[0152] The first generating submodule is used for the execution end to generate the AcScan operator for scanning the acceleration data block according to the operator tree.

[0153] In one embodiment of the present invention, the first calculation module 430 includes:

[0154] A third determining submodule, configured for the execution end to determine, in the server according to the function G, a logical data block corresponding to the acceleration data block through a mapping relationship list;

[0155] A fourth determining submodule, configured for the execution end to determine a Scan operator corresponding to the logical data block according to the logical data block;

[0156] A fifth determining submodule, configured for the execution end to scan and determine the data in the logical data block through the Scan operator;

[0157] The second generating submodule is used for the execution end to convert the format of the data to generate the target acceleration data block.

[0158] In one embodiment of the present invention, it further includes:

[0159] A fourth generating module 440 is configured to, when the query is completed, enable the execution end to perform calculations based on the target acceleration data block or the acceleration data block to generate a calculation result;

[0160] The fifth generating module 450 is configured for the execution end to generate a query result according to the calculation result, and send the query result to the client.

[0161] 5 , there is shown a block diagram of a data query device based on logical data blocks according to an embodiment of the present application;

[0162] A data query device based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The device involves a client, a server, and an execution end, and includes:

[0163] A first constructing module 510 is configured for the server to receive a creation command sent by the database and construct a data model conversion rule according to the creation command;

[0164] A second constructing module 520 is configured for the server to send the data model conversion rule to the database, construct the mapping relationship list based on the logical data block, and send the mapping relationship list to the execution end;

[0165] A second determining module 530 is configured for the server to determine, based on the mapping relationship list, whether the logical data block has the acceleration data block corresponding to the logical data block;

[0166] A second generating module 540 is configured to, when the corresponding acceleration data block does not exist for the logical data block, mark the logical data block as a first data block to be generated, and generate, by the server according to the data model conversion rule, a first newly constructed acceleration block corresponding to the first data block to be generated;

[0167] The first marking module 550 is configured for the server to mark the first newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the first newly constructed acceleration block.

[0168] In one embodiment of the present invention, it further includes:

[0169] a second marking module 560, configured to, when data in the logical data block is modified, mark the acceleration data block corresponding to the logical data block in the mapping relationship list as invalid, and mark all mapping relationships related to the acceleration data block in the mapping relationship list as invalid;

[0170] A third determining module 570 is configured for the server to determine a logical data block corresponding to the acceleration data block using the acceleration data block marked as the failure state;

[0171] A third generating module 580 is configured for the server to mark the logical data block as a second data block to be generated, and the server to generate a second newly constructed acceleration block corresponding to the second data block to be generated according to the data model conversion rule;

[0172] The third marking module 590 is configured for the server to mark the second newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the second newly constructed acceleration block.

[0173] 6 , there is shown a block diagram of a data query device based on logical data blocks according to an embodiment of the present application;

[0174] A data query device based on logical data blocks is used to accelerate data queries in a database using the logical data blocks, wherein the logical data blocks include a plurality of fine-grained logical data blocks obtained by partitioning source data. The device involves a client, a server, and an execution end, and includes:

[0175] A third generating module 610 is configured for the client to generate a query data request according to user requirements and send the query data request to the execution end;

[0176] The first feedback module 620 is configured for the client to receive the query result sent by the execution end and to feed back the query result to the corresponding user.

[0177] 7 , a computer device of a data query method based on a logical data block of the present invention is shown, which may specifically include the following:

[0178] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0179] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0180] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0181] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although not shown in FIG. 7 , a magnetic disk drive may be provided for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.

[0182] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0183] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, camera, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in FIG. 7 , other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.

[0184] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a data query method based on logical data blocks provided by an embodiment of the present invention.

[0185] That is, when the processing unit 16 executes the program, the following is achieved: the execution end receives the query data request sent by the client, and generates an AcScan operator through the optimizer based on the query data request; the execution end scans the acceleration data block in the server based on the AcScan operator, and determines whether the acceleration data block is valid through the scan; when it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on the function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

[0186] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for querying data based on a logical data block as provided in all embodiments of the present application is implemented:

[0187] That is, when the program is executed by the processor, it is implemented as follows: the execution end receives the query data request sent by the client, and generates an AcScan operator through the optimizer based on the query data request; the execution end scans the acceleration data block in the server based on the AcScan operator, and determines whether the acceleration data block is valid through the scan; when it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on the function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalence relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

[0188] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0189] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0190] The computer program code for performing the operations of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0191] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0192] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0193] The above is a detailed introduction to a data query method and device based on logical data blocks provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data query method based on a logical data block, for accelerating a data query in a database through the logical data block, wherein: The logic data block includes a plurality of fine-grained logic data blocks obtained by dividing the source data. The method involves a client, a server and an execution end, and includes: The execution end receives the query data request sent by the client, and generates an AcScan operator through an optimizer according to the query data request; The execution end scans the acceleration data block in the server according to the AcScan operator, and determines whether the acceleration data block is valid through the scanning; When it is determined that the acceleration data block is invalid, the execution end constructs a target acceleration data block based on function G and the mapping relationship list sent by the server, and performs calculations based on the target acceleration data block until the query is completed; wherein, the function G is used to maintain an equivalent relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

2. The method according to claim 1, characterized in that The step of generating an AcScan operator through an optimizer according to the query data request includes: The execution end determines whether to use the acceleration data block in the server by performing a cost evaluation on the query data request according to the optimizer; When it is determined to use the acceleration data block, the execution end determines an operator tree corresponding to the acceleration data block; The execution end generates the AcScan operator for scanning the acceleration data block according to the operator tree.

3. The method according to claim 1, characterized in that The step of constructing a target acceleration data block according to the function G and the mapping relationship list includes: The execution end determines the logic data block corresponding to the acceleration data block in the server through a mapping relationship list according to the function G; The execution end determines a Scan operator corresponding to the logical data block according to the logical data block; The execution end scans and determines the data in the logic data block through the Scan operator; The execution end performs format conversion on the data to generate the target acceleration data block.

4. The method according to claim 1, characterized in that The method further comprises: When the query is completed, the execution end performs calculation according to the target acceleration data block or the acceleration data block to generate a calculation result; The execution end generates a query result according to the calculation result, and sends the query result to the client.

5. A data query method based on a logical data block, for accelerating a data query in a database through the logical data block, wherein: The logic data block is obtained by dividing the source data into a plurality of fine-grained logic data blocks, wherein the method involves a client, a server and an execution end, and the method comprises: The server receives a creation command sent by the database and constructs a data model conversion rule according to the creation command; The server sends the data model conversion rule to the database, constructs the mapping relationship list based on the logical data block, and sends the mapping relationship list to the execution end; The server determines, according to the mapping relationship list, whether the logical data block has the acceleration data block corresponding to the logical data block; When the logical data block does not have a corresponding acceleration data block, the logical data block is marked as a first data block to be generated, and the server generates a first newly constructed acceleration block corresponding to the first data block to be generated according to the data model conversion rule; The server marks the first newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the first newly constructed acceleration block.

6. The method according to claim 5, characterized in that The method further comprises: After the data in the logic data block is modified, the server marks the acceleration data block corresponding to the logic data block as invalid in the mapping relationship list, and All mapping relationships in the mapping relationship list related to the acceleration data block are marked as invalid; The server determines a logical data block corresponding to the acceleration data block through the acceleration data block marked as the failure state; The server marks the logic data block as a second data block to be generated, and generates a second newly constructed acceleration block corresponding to the second data block to be generated according to the data model conversion rule; The server marks the second newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the second newly constructed acceleration block.

7. A data query method based on a logical data block, for accelerating a data query in a database through the logical data block, wherein: The logic data block includes a plurality of fine-grained logic data blocks obtained by dividing the source data. The method involves a client, a server and an execution end, and includes: The client generates a query data request according to user requirements, and sends the query data request to the execution end; The client receives the query result sent by the execution end, and feeds back the query result to the corresponding user.

8. A data query device based on a logical data block, used to accelerate the query of data query in a database through the logical data block, wherein: The logic data block includes a plurality of fine-grained logic data blocks obtained by dividing the source data, and is characterized in that the device involves a client, a server and an execution end, and the device includes: A first generating module, configured for the execution end to receive a query data request sent by the client, and to generate an AcScan operator through an optimizer according to the query data request; A first determination module, configured for the execution end to scan the acceleration data block in the server according to the AcScan operator, and determine whether the acceleration data block is valid through the scanning; The first computing module is used for determining that the acceleration data block is invalid, and the execution end determines that the acceleration data block is invalid according to the function The function G is used to maintain an equivalent relationship between the AcScan operator and the Scan operator corresponding to the logical data block.

9. A data query device based on a logical data block, used to accelerate the query of data query in a database through the logical data block, wherein: The logic data block includes a plurality of fine-grained logic data blocks obtained by dividing the source data, and is characterized in that the device involves a client, a server and an execution end, and the device includes: A first construction module is used for the server to receive a creation command sent by the database and construct a data model conversion rule according to the creation command; A second construction module is used for the server to send the data model conversion rule to the database, and to construct the mapping relationship list based on the logical data block, and to send the mapping relationship list to the execution end; A second determination module, configured for the server to determine whether the logical data block has the acceleration data block corresponding to the logical data block according to the mapping relationship list; A second generating module is configured to mark the logical data block as a first data block to be generated when there is no corresponding acceleration data block for the logical data block, and the server generates a first newly constructed acceleration block corresponding to the first data block to be generated according to the data model conversion rule; The first marking module is used for the server to mark the first newly constructed acceleration block as being in a valid state in the corresponding mapping relationship list according to the first newly constructed acceleration block.

10. A data query device based on a logical data block, used to accelerate the query of data query in a database through the logical data block, wherein: The logic data block includes a plurality of fine-grained logic data blocks obtained by dividing the source data, and is characterized in that the device involves a client, a server and an execution end, and the device includes: A third generating module is used for the client to generate a query data request according to user requirements and send the query data request to the execution end; The first feedback module is used for the client to receive the query result sent by the execution end and The query result is fed back to the corresponding user.

Images