Information delivery method and apparatus, database system, electronic device, and storage medium

Abstract

The application provides an information transmission method and device, a database system, electronic equipment and a storage medium. The method is applied to an execution engine of a database, and the execution engine comprises a scheduler. The method comprises the following steps: obtaining a plurality of operators, wherein the plurality of operators comprise an information collector and an information subscriber; identifying the information collector and the information subscriber in the plurality of operators; establishing a matching relationship between the information subscriber, the scheduler and the information collector respectively; obtaining execution information collected by the information collector; processing the execution information; and sending the processed execution information to the information subscriber and the scheduler matched with the information collector according to a predetermined rule.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to an information transmission method, apparatus, database system, electronic device and storage medium. Background Technology

[0002] With the rapid development of information technology and the explosive growth of data, scenarios involving the processing of massive amounts of data are becoming increasingly common. Structured Query Language (SQL) is a language for communicating with databases, used to read and write data. During the execution of SQL statements in a database, the database execution engine needs to formulate an execution plan and then regulate the execution process based on that plan. In related technologies, the database execution engine estimates the data in the database to generate statistical information and optimizes the execution plan based on this estimated statistical information. However, because the estimated statistical information is inaccurate, the database execution engine may choose an inferior execution plan and operator algorithm, resulting in poor database execution performance. Summary of the Invention

[0003] This application provides an information transmission method, apparatus, database system, electronic device, and storage medium to address the problems existing in related technologies.

[0004] In a first aspect, embodiments of this application provide an information transmission method applied to an execution engine of a database. The execution engine includes a scheduler. The information transmission method includes: obtaining a plurality of operators, wherein the plurality of operators include an information collector and an information subscriber; identifying the information collector and the information subscriber among the plurality of operators; establishing matching relationships between the information subscriber, the scheduler, and the information collector, respectively; acquiring execution information collected by the information collector; processing the execution information; and sending the processed execution information to the information subscriber and the scheduler matched with the information collector according to predetermined rules.

[0005] Secondly, embodiments of this application provide an information transmission device deployed in the execution engine of a database. The execution engine includes a scheduler. The information transmission device includes: an operator acquisition unit configured to acquire multiple operators, wherein the multiple operators include information collectors and information subscribers; an operator identification unit configured to identify the information collectors and information subscribers among the multiple operators; a relationship establishment unit configured to establish matching relationships between the information subscribers, the scheduler, and the information collectors, respectively; an information acquisition unit configured to acquire execution information collected by the information collectors; an information processing unit configured to process the execution information; and an information sending unit configured to send the processed execution information to the information subscribers and the scheduler matched with the information collectors according to predetermined rules.

[0006] Thirdly, embodiments of this application provide a database system including a computing node, the computing node including a scheduler, the computing node being configured to perform the following operations: obtaining a plurality of operators, wherein the plurality of operators include information collectors and information subscribers; identifying the information collectors and information subscribers among the plurality of operators; establishing matching relationships between the information subscribers, the scheduler, and the information collectors respectively; acquiring execution information collected by the information collectors; processing the execution information; and sending the processed execution information to the information subscribers and scheduler matched with the information collectors according to predetermined rules.

[0007] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0008] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0009] Compared with the prior art, this application has the following advantages:

[0010] According to the embodiments of this application, the database execution engine can collect execution information in real time through operators that act as information collectors, and then process and transmit the information to the scheduler and operators that act as information subscribers for plan re-optimization and adaptive selection of operator algorithms. This can effectively improve the efficiency of the database in executing SQL statements and retrieve data that meets user needs more quickly.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0013] Figure 1 This is a schematic diagram illustrating an information transmission scheme according to an embodiment of this application;

[0014] Figure 2 This is a flowchart illustrating an embodiment of an information transmission method according to this application;

[0015] Figure 3 This is a schematic diagram illustrating the execution of an SQL statement according to an embodiment of this application;

[0016] Figure 4 This is a structural block diagram illustrating an information transmission device according to an embodiment of this application; and

[0017] Figure 5 This is a block diagram illustrating an electronic device according to an embodiment of this application. Detailed Implementation

[0018] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0019] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0020] In a prior art application, the database execution engine estimated the statistical information of the data in the database before execution to optimize the execution plan. This art assumed that the data remained unchanged over a period of time and that the old statistical information could accurately reflect the data distribution in the current period. However, there is a large error between the estimated statistical information before execution and the information at the time of execution. Due to the inaccurate estimated statistical information, the database execution engine may choose an inferior execution plan and an inferior operator algorithm, resulting in poor database execution performance.

[0021] In view of this, embodiments of this application provide a scheme for information transmission during database execution to solve all or part of the problems existing in the related technologies. To more clearly illustrate the information transmission scheme provided in the embodiments of this application, the following is combined with... Figure 1 The information transmission scheme shall be explained. Figure 1 This is a schematic diagram illustrating an information transmission scheme according to an embodiment of this application. The information transmission device, as an important component in the database's execution engine, collects execution information collected by operators in the execution layer. This execution information includes at least one of the following: statistical information, data characteristics, and resource usage. It should be noted that the execution information described above is merely exemplary; those skilled in the art can selectively set the content of the execution information based on the technical principles of this application, as long as the principles of this application are implemented. After collecting the execution information, the information transmission device processes the execution information, returns the processed execution message to the scheduler in the scheduling layer for plan re-optimization, and simultaneously returns the processed execution message to the operators in the execution layer for adaptive selection of operator algorithms.

[0022] For example, after receiving an SQL statement, the database execution engine generates an execution plan, which outlines the specific steps the engine takes to execute the SQL statement. Plan re-optimization involves the scheduler re-optimizing the unexecuted execution plan based on processed execution information to obtain a better plan for the unexecuted parts. Even when operators calculate the correct result, different implementation algorithms can lead to performance differences in different scenarios. Operator algorithm adaptive selection refers to the operator dynamically selecting an algorithm using processed execution information to improve its computational performance. Therefore, the information transmission method provided in this application during database execution allows the scheduler and operators to optimize based on accurate execution information, effectively improving database execution efficiency. In the following description, unless otherwise specified, SQL statements will be used as an example of statements received by the execution engine.

[0023] One embodiment of this application provides an information transmission method applied to the execution engine of a database. The execution engine includes a scheduler and can be either a distributed or centralized execution engine. The following will refer to... Figure 2 The information transmission method 200 of this application will be described. Figure 2 This is a flowchart illustrating an embodiment of an information transmission method 200 of this application. For example... Figure 2 As shown, the information transmission method 200 may include the following steps S201 to S206. Steps S201 to S206 will be described in detail below with reference to specific embodiments.

[0024] First, proceed to step S201. In step S201, multiple operators are obtained, wherein the multiple operators include information collectors and information subscribers.

[0025] Taking SQL statements as an example, SQL statements are the most common method for manipulating databases and data within them. Through a series of keywords defined in the SQL standard, database users can manage and manipulate the data within the database. An SQL statement typically includes one or more operators, each responsible for performing a basic data processing logic. Basic operators include scan, join, aggregation, and filter operators.

[0026] In one embodiment, for example, the SQL statement received by the database execution engine is SELECT * FROM T1 JOINT2 ON T1.id=T2.id WHERE T2.num>100. From this SQL statement, it can be seen that the database contains a data table named T1 and a data table named T2. The operators obtained from this SQL statement include a join operator, a scan operator for table T1, a scan operator for table T2, and a WHERE operator. Among these operators, there are both information collectors and information subscribers. Information collectors gather runtime execution information during the execution of the SQL statement, and information subscribers receive the execution information collected by the information collectors to adaptively select operator algorithms.

[0027] Next, proceed to step S202. In step S202, identify the information collector and the information subscriber among the plurality of operators.

[0028] In one embodiment, an SQL statement performs a join operation on two tables (i.e., the SQL statement contains a join operator). Assume one table is large (i.e., has a large amount of data) and the other is small (i.e., has a small amount of data). The scan operator on the small table can be identified as an information collector, and the scan operator on the large table can be identified as an information subscriber. Taking the SQL statement `SELECT * FROM T1 JOIN T2 ON T1.id=T2.id WHERE T2.num>100` as an example, assume that the data volume of table T1 is greater than that of table T2, for example, table T1 stores 100,000 records and table T2 stores 3,000 records. The following will refer to... Figure 3 The execution process of this SQL statement will be explained. Figure 3 This is a schematic diagram illustrating the execution of an SQL statement according to an embodiment of this application. Figure 3 As shown, the `where` operator first filters data in table T2 based on the condition `num > 100`. Then, the `scan` operator scans the data in table T2 that meets the `num > 100` filter condition. Next, the `scan` operator scans all data in table T1. Finally, the `join` operator joins the data in table T1 and table T2 that meet the condition `T1.id = T2.id`. In this process, the entire data in table T1 needs to be transferred to the `join` operator during the `join` operation. Because the data volume in table T1 is much larger than that in table T2, most of the data in table T1 will not be selected by the `join` operator, making this process inefficient. If, before the `join` operation, the `scan` operator of table T2 passes the execution information to the `scan` operator of table T1 in advance, the `scan` operator of table T1 can filter the data in table T1 in advance based on the execution information passed from the `scan` operator of table T2, thus significantly improving the efficiency of this process. Therefore, the scan operator for table T2 is identified as an information collector, and the scan operator for table T1 is identified as an information subscriber.

[0029] In one embodiment, an SQL statement queries data from a table based on multiple predetermined conditions (i.e., the SQL statement contains a WHERE operator). The table's scan operator can be identified as an information collector, and the table's WHERE operator can be identified as an information subscriber. Taking the SQL statement `SELECT * FROM T1 WHERE T1.a>1 and T1.b>2 and T1.c>3` as an example, the WHERE operator of table T1 needs to filter the data in table T1 according to the predetermined conditions (i.e., T1.a>1 and T1.b>2 and T1.c>3). For example, if table T1 has 10,000 records, the scan operator of table T1, during its scan, determines, based on the data distribution in column c of table T1, that the condition `T1.c>3` can filter out 99.9% of the data. Therefore, only 10 records satisfy the condition `T1.c>3`. If we execute T1.c>3 first, then T1.a>1 and T1.b>2, then only 10 records need to be processed for T1.a>1 and T1.b>2, which can greatly improve the database's execution efficiency. The more data a predefined condition can filter out, the higher it ranks in the order. Therefore, after processing the execution information, the order of the predefined conditions changes from the original order T1.a>1 and T1.b>2 and T1.c>3 to T1.c>3 and T1.a>1 and T1.b>2. The scan operator of table T1 sends the collected execution information to the WHERE operator of table T1, optimizing the efficiency of the WHERE operator in filtering data. Therefore, the scan operator of table T1 is identified as the information collector, and the WHERE operator is identified as the information subscriber.

[0030] Those skilled in the art should understand that the SQL statements and their execution process described above are merely exemplary and are not intended to limit this application. Those skilled in the art can select information collectors and information subscribers from among multiple operators based on the technical principles of this application, as long as the technical principles of this application are implemented.

[0031] Next, proceed to step S203. In step S203, a matching relationship is established between the information subscriber, the scheduler, and the information collector, respectively.

[0032] The database execution engine needs to establish matching relationships between information subscribers and information collectors, as well as between the scheduler and information collectors, to ensure that the information collectors deliver the collected execution information to their matched information subscribers and schedulers. The scheduler's role is to break down SQL statements into multiple subtasks for parallel processing. In one embodiment, the matching relationships between information subscribers and information collectors, and between the scheduler and information collectors, can be established using a JSON (JavaScript Object Notation) data structure with key-value pairs. For example, in establishing the matching relationship between information subscribers and information collectors, the information collector is stored as the key and the information subscriber as the value; similarly, in establishing the matching relationship between the scheduler and information collectors, the information collector is stored as the key and the scheduler as the value. This explanation is merely illustrative and describes the matching relationships between information subscribers and information collectors, the matching relationships between the scheduler and information collectors, and the process of establishing such matching relationships. The technical principles of this application can also cover existing or future developments of such matching relationships.

[0033] Next, proceed to step S204. In step S204, obtain the execution information collected by the information collector.

[0034] In one embodiment, taking the SQL statement SELECT * FROM T1 JOIN T2 ON T1.id=T2.id WHERET2.num>100 as an example, in this SQL statement, the scan operator of the T2 data table is the information collector. During the process of scanning the data, the scan operator of the T2 data table collects execution information, which may include, for example, the number of rows in the T2 data table, the number of different values ​​in a certain column, the maximum value, the minimum value, etc.

[0035] In some embodiments, the execution information includes at least one of the following: statistical information, data characteristics, and resource usage. Statistical information may be information such as the number of rows in a data table, the number of distinct values ​​in a column, the maximum value, the minimum value, etc. Data characteristics may be information such as range, variance, standard deviation, etc. Resource usage may be information such as storage resource utilization rate, computing resource utilization rate, etc.

[0036] Next, proceed to step S205. In step S205, the execution information is processed.

[0037] In one embodiment, the database execution engine obtains execution information from the information collector consisting of all unique values ​​in a certain column. For example, using the SQL statement `SELECT * FROM T1 JOIN T2 ON T1.id=T2.id WHERE T2.num>100`, the database execution engine obtains the execution information as all unique values ​​in the `id` column of table `T2`. The database execution engine then processes this execution information to generate information that information subscribers and the scheduler can use to optimize execution efficiency.

[0038] In one embodiment, processing the execution information includes: creating a binary array; selecting at least one hash function; and mapping the execution information to the binary array using the at least one hash function to generate a Bloom filter.

[0039] In some embodiments, the database execution engine can create binary arrays. An array is a collection used to store a set of data of the same type; therefore, a binary array is a collection used to store a set of binary data. Binary data indicates that the data can only be 0 or 1. After creating the binary array, the database execution engine sets all data in the array to 0. For example, the database execution engine creates a binary array of length 8, with an initial data state of {0,0,0,0,0,0,0,0}.

[0040] After creating the binary array, the database execution engine can select at least one hash function. A hash function can transform an input value of arbitrary length into a fixed-length output value, called a hash value. Hash functions are one-way; the original data cannot be derived from the hash value, thus ensuring the security of the hash function input value. The database execution engine selects at least one hash function for subsequent hash operations.

[0041] The database execution engine can map the execution information to the binary array using at least one hash function to generate a Bloom filter. Taking the SQL statement `SELECT * FROM T1 JOIN T2 ON T1.id=T2.id WHERE T2.num>100` as an example, the database execution engine selects a hash function `Hash1`. The execution information obtained by the database execution engine is all the deduplicated values ​​in the `id` column of the `T2` data table, such as 110, 120, and 121. The database execution engine inputs the obtained execution information into `Hash1` and takes the hash value modulo the length of the binary array. For example, if the length of the binary array is 8, inputting 110 into `Hash1` and taking the modulo of 8 results in 3, i.e., `Hash1(110) % 8 = 3`. Mapping 3 to the index of the binary array and setting the data corresponding to that index to 1 updates the binary array from the original {0,0,0,0,0,0,0,0} to {0,0,0,1,0,0,0,0}. A Bloom filter is created by mapping the binary array of execution information using a hash function. The Bloom filter determines whether a piece of data exists in a set; 0 indicates the data does not exist in the set, and 1 indicates it does. After generating the Bloom filter, the database execution engine can pass it to the information subscriber, i.e., the scan operator of table T1, so that the information subscriber can filter the data before joining, improving execution efficiency. The choice of Bloom filter described above is merely exemplary; those skilled in the art can choose filters that can be used for information consumers based on actual circumstances, as long as the principle of this embodiment is achieved.

[0042] In another embodiment, the database execution engine can generate a histogram based on execution information collected by the information collector operator. The histogram can include a frequency histogram and a TOP-K histogram. A frequency histogram represents the frequency of occurrence of each value in a column, where the horizontal axis represents sequentially arranged intervals, and its height represents the number of times (frequency) the data appears within that interval. A TOP-K histogram displays the K most frequent values ​​in a column.

[0043] In one embodiment, processing the execution information includes: determining whether the operator corresponding to the information collector is distributed across different data segments. If the operator corresponding to the information collector is distributed across different data segments, the execution information is aggregated into global information. In the case of massive amounts of data, operators are split across different machines to process a portion of the global data, thereby achieving concurrent execution of operators and improving the execution efficiency of SQL statements. If the execution information received by the database execution engine is the execution information collected by an operator for a portion of the entire dataset, then the operator is distributed across different data segments. For example, consider the SQL statement SELECT * FROM T1 JOIN T2 ON T1.id = T2.id WHERE T2.num > 100, where table T2 stores 3000 records. First, determine whether the operators corresponding to the information collector are distributed across different data segments. During the scanning of the T2 data table, in order to improve scanning performance, the T2 data table can be split into 3 data segments, each with 1000 data entries. The scan operators corresponding to the information collector and these three data segments are then run on 3 different machines. In this scenario, the operators corresponding to the information collectors are distributed across three different data segments. Therefore, each scan operator can only scan one data segment. For example, the execution information collected by the scan operator on the first data segment is all the unique values ​​of the id column in the T2 data table after deduplication: 100, 101, 102. The execution information collected by the scan operator on the second data segment is all the unique values ​​of the id column in the T2 data table after deduplication: 110, 111, 112. The execution information collected by the scan operator on the third data segment is all the unique values ​​of the id column in the T2 data table after deduplication: 120, 121, 122. At this point, the database execution engine needs to aggregate the above three local execution information into global execution information, that is, aggregate all the unique values ​​of the id column in the T2 data table on the above three data segments into 100, 101, 102, 110, 111, 112, 120, 121, 122.

[0044] Next, proceed to step S206. In step S206, the processed execution information is sent to the information subscribers and schedulers that match the information collector according to predetermined rules.

[0045] After processing the execution information, the database execution engine needs to send the processed information to the information subscribers and scheduler matched with the information collectors. In one embodiment, the database execution engine can send a Bloom filter to the information consumer, allowing the consumer to select a better algorithm based on the information in the Bloom filter, thus improving the operator's execution efficiency. In another embodiment, the database execution engine can send a histogram to the scheduler, enabling the scheduler to re-optimize the unexecuted parts of the execution plan at a more macroscopic level, resulting in a better plan for the unexecuted portion and improving execution efficiency.

[0046] In one embodiment, sending the processed execution information to an information subscriber and a scheduler that match the information collector according to predetermined rules includes: establishing a broadcast channel with the information subscriber; and sending the processed execution information to the information subscriber through the broadcast channel.

[0047] In some embodiments, when dealing with massive amounts of data, the operator acting as an information subscriber can be split across different machines to achieve concurrent execution of the operator, thereby improving the execution efficiency of SQL statements. Therefore, it is necessary to establish a broadcast channel with multiple information subscribers. This channel transmits the processed execution data to the information subscribers with low transmission overhead, saving the high communication costs associated with establishing multiple transmission channels for multiple information subscribers. These multiple information subscribers are matched with a specific information collector. For example, consider the SQL statement `SELECT * FROM T1 JOIN T2 ON T1.id=T2.id WHERE T2.num>100`. Table T1 has 100,000 rows of data, and its scan operator is an information subscriber. During the scan of table T1, to improve scanning performance, it can be split into 20 data segments, representing 20 information subscribers. These 20 information subscribers are matched with the information collector (i.e., the scan operator of table T2). At this point, the database execution engine only needs to establish one broadcast channel connected to the 20 information subscribers, instead of establishing 20 separate transmission channels, which greatly reduces the transmission overhead of execution information.

[0048] In some embodiments, after a broadcast channel is established, the database's execution engine sends the processed execution information to the broadcast channel, and then all information subscribers connected to the broadcast channel can receive the processed execution information through the broadcast channel.

[0049] In one embodiment, the processed execution information is sorted according to predetermined conditions. Taking the SQL statement `SELECT * FROM T1 WHERE T1.a>1 and T1.b>2 and T1.c>3` as an example, the WHERE operator in table T1 needs to filter the data in table T1 according to predetermined conditions (i.e., T1.a>1 and T1.b>2 and T1.c>3). For example, if table T1 has 10,000 records, and the condition `T1.c>3` can filter out 99.9% of the data, then only 10 records satisfy the condition `T1.c>3`. Therefore, `T1.c>3` should be executed first, followed by `T1.a>1 and T1.b>2`. This way, only 10 records need to be processed in total for `T1.a>1 and T1.b>2`, which can greatly improve the database execution efficiency. The more data a predefined condition can filter out, the higher it ranks in the order. Therefore, after processing the execution information, the predefined condition sorting is changed from the original sorting T1.a>1 and T1.b>2 and T1.c>3 to the sorting T1.c>3 and T1.a>1 and T1.b>2. The database execution engine then sends this sorted data according to the predefined conditions to the information subscribers so that they can perform the data filtering process more efficiently.

[0050] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an information transmission device 400, an execution engine deployed in a database, the execution engine including a scheduler, which will be referred to below. Figure 4 The information transmission device 400 of this application will be described. Figure 4 This is a structural block diagram illustrating an information transmission device according to an embodiment of this application. Figure 4 As shown, the information transmission device 400 may include: an operator acquisition unit 401, an operator identification unit 402, a relationship establishment unit 403, an information acquisition unit 404, an information processing unit 405, and an information sending unit 406.

[0051] The operator acquisition unit 401 is configured to acquire multiple operators, including information collectors and information subscribers.

[0052] The operator acquisition unit 401 may be a component in the database execution engine, through which the database execution engine acquires multiple operators, including information collectors and information subscribers.

[0053] In one embodiment, for example, the SQL statement received by the database execution engine is SELECT * FROM T1 JOINT2 ON T1.id=T2.id WHERE T2.num>100. From this SQL statement, it can be seen that the database contains a data table named T1 and a data table named T2. The operators obtained from this SQL statement include a join operator, a scan operator for table T1, a scan operator for table T2, and a WHERE operator. Among these operators, there are both information collectors and information subscribers. Information collectors gather runtime execution information during the execution of the SQL statement, and information subscribers receive the execution information collected by the information collectors to adaptively select operator algorithms.

[0054] Operator identification unit 402 is configured to identify the information collector and the information subscriber among the plurality of operators.

[0055] The operator identification unit 402 can be a component in the database's execution engine, which uses this component to identify the information collector and the information subscriber among the plurality of operators.

[0056] In one embodiment, an SQL statement performs a join operation on two tables (i.e., the SQL statement contains a join operator). Assume one table is large (i.e., has a large amount of data) and the other is small (i.e., has a small amount of data). The scan operator on the small table can be identified as an information collector, and the scan operator on the large table can be identified as an information subscriber. Taking the SQL statement `SELECT * FROM T1 JOIN T2 ON T1.id=T2.id WHERE T2.num>100` as an example, assume that the data volume of table T1 is greater than that of table T2, for example, table T1 stores 100,000 records and table T2 stores 3,000 records. The following will refer to... Figure 3 The execution process of this SQL statement will be explained. Figure 3 This is a schematic diagram illustrating the execution of an SQL statement according to an embodiment of this application. Figure 3As shown, the `where` operator first filters data in table T2 based on the condition `num > 100`. Then, the `scan` operator scans the data in table T2 that meets the `num > 100` filter condition. Next, the `scan` operator scans all data in table T1. Finally, the `join` operator joins the data in table T1 and table T2 that meet the condition `T1.id = T2.id`. In this process, the entire data in table T1 needs to be transferred to the `join` operator during the `join` operation. Because the data volume in table T1 is much larger than that in table T2, most of the data in table T1 will not be selected by the `join` operator, making this process inefficient. If, before the `join` operation, the `scan` operator of table T2 passes the execution information to the `scan` operator of table T1 in advance, the `scan` operator of table T1 can filter the data in table T1 in advance based on the execution information passed from the `scan` operator of table T2, thus significantly improving the efficiency of this process. Therefore, the scan operator for table T2 is identified as an information collector, and the scan operator for table T1 is identified as an information subscriber.

[0057] The relationship establishment unit 403 is configured to establish matching relationships between the information subscriber, the scheduler, and the information collector, respectively.

[0058] The relationship establishment unit 403 can be a component in the database's execution engine, through which the database's execution engine establishes matching relationships between the information subscriber, the scheduler, and the information collector, respectively.

[0059] The database execution engine needs to establish matching relationships between information subscribers and information collectors, and between the scheduler and information collectors, so that the information collectors can pass the collected execution information to their corresponding counterparts. The scheduler's role is to break down SQL statements into multiple subtasks for parallel processing. In one embodiment, the matching relationships between information subscribers and information collectors, and between the scheduler and information collectors, can be established using a JSON (JavaScript Object Notation) data structure in key-value pairs. For example, in establishing the matching relationship between information subscribers and information collectors, the information collector is stored as the key and the information subscriber as the value; in establishing the matching relationship between the scheduler and information collectors, the information collector is stored as the key and the scheduler as the value.

[0060] The information acquisition unit 404 is configured to acquire the execution information collected by the information collector.

[0061] The information acquisition unit 404 can be a component in the database's execution engine, through which the database's execution engine acquires the execution information collected by the information collector.

[0062] Information processing unit 405 is configured to process the execution information.

[0063] The information processing unit 405 can be a component in the database's execution engine, through which the database's execution engine processes the execution information.

[0064] The information sending unit 406 is configured to send the processed execution information to the information subscribers and schedulers that match the information collector according to a predetermined rule.

[0065] The information sending unit 406 can be a component in the database's execution engine, through which the database's execution engine sends the processed execution information to the information subscribers and schedulers that match the information collector according to predetermined rules.

[0066] This application provides a database system including a computing node, the computing node including a scheduler, the computing node being configured to perform the following operations: obtaining a plurality of operators, wherein the plurality of operators include information collectors and information subscribers; identifying the information collectors and information subscribers among the plurality of operators; establishing matching relationships between the information subscribers, the scheduler, and the information collectors respectively; acquiring execution information collected by the information collectors; processing the execution information; and sending the processed execution information to the information subscribers and scheduler matched with the information collectors according to predetermined rules.

[0067] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0068] According to the embodiments of this application, the database execution engine can collect execution information in real time during the execution process through the information collector operator, and then process the information and pass it to the scheduler and information subscriber operator for plan re-optimization and adaptive selection of operator algorithms. This can effectively improve the efficiency of the database in executing SQL statements and retrieve data that meets user needs more quickly.

[0069] Figure 5 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 5As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can run on the processor 502. When the processor 502 executes the computer program, it implements the method described in the above embodiments. The number of memories 501 and processors 502 can be one or more.

[0070] The electronic device also includes:

[0071] Communication interface 503 is used to communicate with external devices and perform data exchange and transmission.

[0072] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0073] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0074] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0075] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0076] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0077] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0078] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0079] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0080] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0081] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0083] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0084] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0086] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An information transmission method applied to the execution engine of a database, the execution engine including a scheduler, the information transmission method comprising: Multiple operators are obtained, wherein the multiple operators include an information collector and an information subscriber; the information collector is used to collect execution information during the execution process, and the information subscriber is used to receive the execution information for adaptive selection of operator algorithms; Identify the information collector and the information subscriber among the plurality of operators; Establish matching relationships between the information subscribers, the scheduler, and the information collectors, respectively; Obtain the execution information collected by the information collector; wherein the execution information includes at least one of the following: statistical information, data characteristics, and resource usage; The execution information is processed, the processing including: creating a binary array, selecting at least one hash function, and mapping the execution information to the binary array through the at least one hash function to generate a Bloom filter; and The processed execution information is sent to the information subscriber and scheduler that are matched with the information collector according to a predetermined rule; wherein the predetermined rule includes: establishing a broadcast channel with the information subscriber, and sending the processed execution information to the information subscriber through the broadcast channel.

2. The method according to claim 1, wherein, The processing of the execution information includes: Determine whether the operators corresponding to the information collector are distributed across different data segments, wherein, If the operators corresponding to the information collector are distributed across different data segments, then the execution information is aggregated into global information.

3. The method according to claim 1, wherein, The processed execution information is sorted according to predetermined conditions.

4. An information transmission device, deployed in the execution engine of a database, the execution engine including a scheduler, the information transmission device comprising: An operator acquisition unit is configured to acquire multiple operators, wherein the multiple operators include an information collector and an information subscriber; the information collector is used to collect execution information during the execution process, and the information subscriber is used to receive the execution information for adaptive selection of operator algorithms; An operator identification unit is configured to identify the information collector and the information subscriber among the plurality of operators; A relationship-establishing unit is configured to establish matching relationships between the information subscriber, the scheduler, and the information collector, respectively. An information acquisition unit is configured to acquire execution information collected by the information collector; wherein the execution information includes at least one of the following: statistical information, data characteristics, and resource usage; An information processing unit is configured to process the execution information, the processing including: creating a binary array, selecting at least one hash function, and mapping the execution information to the binary array through the at least one hash function to generate a Bloom filter; and An information sending unit is configured to send processed execution information to information subscribers and schedulers that match the information collector according to predetermined rules; wherein, the predetermined rules include: establishing a broadcast channel with the information subscriber and sending the processed execution information to the information subscriber through the broadcast channel.

5. A database system comprising a compute node, the compute node including a scheduler, the compute node being configured to perform the following operations: Multiple operators are obtained, among which, The multiple operators include information collectors and information subscribers; The information collector is used to collect execution information during the execution process, and the information subscriber is used to receive the execution information for adaptive selection of operator algorithms; Identify the information collector and the information subscriber among the plurality of operators; Establish matching relationships between the information subscribers, the scheduler, and the information collectors, respectively; Obtain the execution information collected by the information collector; wherein the execution information includes at least one of the following: statistical information, data characteristics, and resource usage; The execution information is processed, the processing including: creating a binary array; selecting at least one hash function; and mapping the execution information to the binary array using the at least one hash function to generate a Bloom filter; and The processed execution information is sent to the information subscriber and scheduler that are matched with the information collector according to a predetermined rule; wherein the predetermined rule includes: establishing a broadcast channel with the information subscriber, and sending the processed execution information to the information subscriber through the broadcast channel.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-3.

7. A computer-readable storage medium storing a computer program therein, which, when executed by a processor, implements the method of any one of claims 1-3.

Images