Performance diagnosis method of data warehouse and related equipment thereof

Abstract

The invention relates to a performance diagnosis method of a data warehouse and related equipment thereof, the data warehouse comprises a master node and a slave node, the method comprises the steps that collected execution information is stored in a disk of the slave node, and the execution information is information generated in the process that the slave node executes a query task of the data warehouse; and in response to an access request for the execution information, sending the execution information read from the disk to the main node so as to aggregate the execution information at the main node to obtain processing information, and determining the performance of the data warehouse according to the processing information. The problem that the query performance of the data warehouse is reduced due to the fact that a diagnosis system of the data warehouse causes large extra overhead on the data warehouse is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a performance diagnosis method for a data warehouse and related equipment. Background Art

[0002] A data warehouse is a data storage system used to support decision analysis and business intelligence, and is often used for large-scale data analysis and query. In a data warehouse, query execution performance is the most obvious indicator for users. When analytical queries execute slowly or abnormally, in order to locate the performance bottleneck or abnormality of the data warehouse, a diagnostic system is needed to accurately and efficiently collect query execution information. The execution information includes running indicators, which include query execution time, hardware resources occupied, number of data rows processed, etc., so that the performance bottleneck or abnormality of the data warehouse can be located based on this execution information through the diagnostic system.

[0003] The MPP architecture is a computing mode for processing analytical tasks (MPP stands for Massively Parallel Processing). This mode distributes computing tasks to multiple server nodes, and after each server node completes the calculation, the respective calculation results are aggregated to obtain the final result. A common MPP architecture for data warehouses is the master-slave mode, which divides the server nodes of the data warehouse into leader nodes and follower nodes. The leader node is used to distribute computing tasks, and the follower node is used to store data and execute computing tasks. In the distributed data warehouse of the MPP architecture, the complexity and operating overhead of the distributed data warehouse will increase accordingly under the premise of equal capabilities. Since the query of the data warehouse will be executed in parallel on multiple distributed slave nodes, the diagnostic system needs to collect the execution information of the query on multiple slave nodes at the same time and merge and store the execution information after transmitting the execution information to the master node or other monitoring system. At the same time, under the MPP architecture, the diagnostic system inevitably needs to face the performance problem of large amounts of data transmission over the network. In this context, the performance of the diagnostic system in terms of accuracy and efficiency becomes particularly important. Since the diagnostic system needs to transmit the execution information of the data warehouse query over the network each time it collects the execution information, it will cause a large additional overhead to the data warehouse, increase the performance impact on the data warehouse query link, reduce the query performance of the data warehouse, and affect the user experience.

[0004] Currently, no effective solution has been proposed to address the problem of reducing data warehouse query performance in related technologies. Summary of the invention

[0005] The performance diagnosis method of a data warehouse and related devices provided by the embodiments of the present invention at least solve the problem that the diagnosis system may cause large additional overhead to the data warehouse, thereby reducing the query performance of the data warehouse.

[0006] According to one aspect of the present invention, a performance diagnosis method for a data warehouse is provided, wherein the data warehouse includes a master node and a slave node, and the method includes:

[0007] The collected execution information is stored in the disk of the slave node, wherein the execution information is information generated during the process of the slave node executing the query task of the data warehouse;

[0008] In response to an access request for the execution information, the execution information read from the disk is sent to the master node, so that the execution information is aggregated at the master node to obtain processing information, and the performance of the data warehouse is determined based on the processing information.

[0009] In some embodiments, before the master node aggregates the execution information to obtain processing information, the method further includes:

[0010] The execution information is stored in a predetermined memory of the slave node, wherein the predetermined memory is a memory shared by execution processes executing different query tasks on the slave node;

[0011] Then, before the master node aggregates the execution information to obtain processing information, the method further includes:

[0012] In response to an access request for the execution information, the execution information read from the predetermined memory is sent to the master node.

[0013] In some embodiments, the step of storing the execution information in a predetermined memory of the slave node includes:

[0014] Creating a hash table in the predetermined memory to store the execution information in the hash table;

[0015] Then, the step of storing the collected execution information in the disk of the slave node includes:

[0016] The execution information read from the hash table is written to the disk.

[0017] In some embodiments, a hash table is created in the predetermined memory, the query task includes multiple operators, and after the slave node executes the operator, execution sub-information is generated, the execution information includes the execution sub-information of the multiple operators, and the step of storing the execution information in the predetermined memory of the slave node further includes:

[0018] Acquire the collected execution sub-information, and determine the identity information of the operator corresponding to the execution sub-information;

[0019] The execution sub-information is stored in the hash table based on the identity information, wherein the storage structure of the hash table includes a key Key and a value Value, the Key is the identity information, and the Value is the execution sub-information.

[0020] In some embodiments, the step of storing the execution sub-information in the hash table based on the identity information includes:

[0021] Collecting update information of the execution sub-information, where the update information is information generated when the slave node executes the operator to a predetermined execution node;

[0022] Determine, according to the identity information, a target Value in the hash table for storing the execution sub-information corresponding to the operator;

[0023] The target Value is updated based on the update information.

[0024] In some embodiments, the step of updating the execution sub-information stored in the target Value based on the update information includes:

[0025] After setting a row-level lock at the target Value of the hash table, the execution sub-information stored in the target Value is updated based on the update information.

[0026] In some embodiments, before determining the performance of the data warehouse according to the processing information, the method further includes:

[0027] Pre-aggregating the execution sub-information at the slave node to obtain pre-processing information;

[0028] The pre-processing information is sent to the master node, so that the master node aggregates the pre-processing information to obtain the processing information.

[0029] According to another aspect of the present invention, a performance diagnosis system for a data warehouse is provided, wherein the diagnosis system is used to execute the performance diagnosis method.

[0030] According to another aspect of the present invention, a data warehouse is provided, wherein the data warehouse includes the performance diagnosis system.

[0031] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor, and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the performance diagnosis method of the data warehouse.

[0032] According to another aspect of the present invention, a non-transitory machine-readable medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the data warehouse performance diagnosis method.

[0033] Beneficial effects of the embodiments of the present invention:

[0034] The embodiment of the present invention can adopt a distributed information maintenance mode in the process of executing query tasks in parallel for multiple slave nodes, store the collected execution information of the query tasks on the disk of the corresponding slave node, and maintain the execution information of the query tasks currently executed by the slave node at each slave node. When the execution information stored and counted by the slave node is accessed, the execution information read from the disk is sent to the master node, so that the execution information is aggregated at the master node. In this way, the network transmission overhead generated by collecting the execution information is amortized from the information collection stage to the execution information access stage. In this way, after the slave node executes the query task, it only needs to perform a disk write operation on the execution information locally, and does not need to transmit data in the network. Therefore, the network transmission overhead can be greatly reduced, thereby improving the query performance of the data warehouse.

[0035] The details of one or more embodiments of the invention are set forth in the following drawings and description so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0037] Figure 1 A schematic diagram of a flow chart of a performance diagnosis method for a data warehouse provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a process for storing execution information in a predetermined memory of a slave node according to an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of a process for storing operator execution sub-information through a hash table provided in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a process for updating execution sub-information in a hash table provided by an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a process for aggregating execution sub-information provided by an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of a performance diagnosis system provided by an embodiment of the present invention storing execution sub-information in a hash table to a disk. DETAILED DESCRIPTION

[0043] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0044] In the data warehouse of the mainstream MPP architecture, the mode of collecting and maintaining the execution information of the query task is usually to collect the execution information on the slave node, and after the query task is executed, the statistical execution information is collected to the master node or transmitted to other monitoring systems, and then the execution information is merged and analyzed at the master node, and finally the execution information is stored on the disk for access by the external data consumer. After in-depth research, the inventor of the present invention found that although the above-mentioned mode of collecting and maintaining execution information can ensure the real-time and accuracy of the execution information, since the execution information needs to be transmitted via the network after each query, it will cause a large additional overhead to the data warehouse, which will have a certain impact on the performance of the data warehouse and reduce the query performance of the data warehouse. Secondly, since the processing and maintenance of the execution information are centralized, the concurrency and scalability are insufficient. When the number of server nodes in the data warehouse increases linearly, the load overhead of the master node or the slave node storing the execution information will increase accordingly, affecting the performance of the data warehouse. In addition, in order to provide the ability to monitor the real-time status of the query, the execution information of the query task needs to be collected to the master node at a certain period, and due to the overhead caused by network transmission, this solution is difficult to adopt a higher data update frequency. This limitation will affect the real-time and accuracy of the query real-time status data.

[0045] In this regard, the first embodiment of the present invention provides a performance diagnosis method for a data warehouse, which is executed by a performance diagnosis system for a data warehouse, wherein the data warehouse includes a master node and a slave node, such as Figure 1 As shown, the performance diagnosis system includes the following steps when executing the method:

[0046] Step S11: Store the collected execution information in the disk of the slave node. The execution information is the information generated in the process of the slave node executing the query task of the data warehouse. The query task executed by the slave node is dispatched by the master node. The execution information of the query task includes the total execution time of the query task, memory usage, disk read data volume, network transmission data volume, number of output data rows, etc.

[0047] Step S12: In response to an access request for execution information, the execution information read from the disk is sent to the master node, so that the execution information is aggregated at the master node to obtain processing information, and the performance of the data warehouse is determined based on the processing information.

[0048] The embodiment of the present invention stores the collected execution information of query tasks executed by each slave node on the disk of the corresponding slave node. When the execution information needs to be accessed, the execution information read from the disk is sent to the master node for aggregation processing, thereby calculating the total execution time, total memory usage and other processing information of the selected query task, and determining the performance of the data warehouse based on the obtained processing information.

[0049] Therefore, the embodiment of the present invention can adopt a distributed information maintenance mode in the process of executing query tasks in parallel for multiple slave nodes, store the collected execution information of the query tasks on the disk of the corresponding slave node, and maintain the execution information of the query tasks currently executed by the slave node at each slave node. When the execution information stored in the statistics of the slave node is accessed, the execution information read from the disk is sent to the master node, so that the execution information is aggregated at the master node. In this way, the network transmission overhead generated by collecting the execution information is amortized from the information collection stage to the execution information access stage. In this way, after the slave node executes the query task, it only needs to perform a disk write operation on the execution information locally, and does not need to transmit data in the network. Therefore, the network transmission overhead can be greatly reduced, thereby improving the query performance of the data warehouse.

[0050] In step S12, before the master node aggregates the execution information to obtain the processing information, Figure 2As shown, the method provided by the embodiment of the present invention further includes: storing the execution information in a predetermined memory of the slave node, wherein the predetermined memory is a memory shared by the execution processes executing different query tasks on the slave node. Then in step S12, before the master node aggregates the execution information to obtain the processing information, the method provided by the embodiment of the present invention further includes: responding to the access request for the execution information, sending the execution information read from the predetermined memory to the master node. Therefore, in addition to storing the execution information through a disk, the embodiment of the present invention can also maintain the execution information through a predetermined memory shared by multiple execution processes on the slave node. Since the predetermined memory has a faster read and write speed for the execution information, the disk read and write overhead can be reduced, thereby further improving the query performance of the data warehouse, while enhancing the real-time and accuracy of the stored execution information, and realizing the real-time data of the execution information being saved through the predetermined memory. When the user needs to access the real-time data of the query task, the execution information can be obtained from the predetermined memory, thereby accurately diagnosing the performance of the data warehouse. In addition, since the execution information is directly read and written in the predetermined memory without the need for file system operations, it can also alleviate problems such as lock competition caused by concurrent read and write operations of the file system, thereby improving the concurrent performance of each server node of the data warehouse.

[0051] Before executing a query task from a node in an embodiment of the present invention, the slave node will record the first metadata of the query task (such as the identity ID of the query task, user, queried warehouse, start time, etc.) in a shared hash table created separately in a predetermined memory, so that all execution processes of the slave node can access the corresponding query task.

[0052] Among them, the step of storing the execution information in the predetermined memory of the slave node includes: creating a hash table in the predetermined memory to store the execution information in the hash table. The embodiment of the present invention uses a hash table to efficiently store the execution information, which can achieve high-frequency maintenance of the execution information, reduce the cost of retrieving the execution information, and has high concurrency. Therefore, the embodiment of the present invention can adopt a higher maintenance frequency for the execution information under the premise of reducing the overhead as much as possible to ensure the real-time and accuracy of the execution information. Then in step S11, the step of storing the collected execution information in the disk of the slave node includes: writing the execution information read from the hash table to the disk. It can be seen that when the embodiment of the present invention performs a disk write operation on the execution information, the execution information stored in the hash table in the predetermined memory can be directly read out and stored in the disk. When the user needs to access the execution information of the historical query task later, the execution information of the historical query task can be read from the disk. And for the execution information stored in the disk, it can be deleted from the hash table, further improving the real-time maintenance capability of the hash table for other execution information.

[0053] It can be seen that the embodiment of the present invention reduces the maintenance overhead of real-time data by using a predetermined memory combined with a hash table to maintain the real-time data of the execution information, and also reduces the impact on the query performance of the data warehouse when storing the execution information. A higher data update frequency can be used to maintain the execution information, thereby enhancing the real-time and accuracy of the execution information. In addition, the embodiment of the present invention stores the execution information based on a hash table and a disk, and supports access to the real-time data of the query task execution information and the execution information of the historical query tasks. Specifically, when it is necessary to access the real-time data of the execution information, the real-time data can be read from the hash table of the predetermined memory of the corresponding slave node, and when accessing the execution information of the historical query task, the execution information of the historical query task can be read from the disk.

[0054] A query task is usually divided into multiple operators, such as operators for data scanning, data aggregation, data association, data transmission, etc. In the embodiment of the present invention, when a query task includes multiple operators, execution sub-information is generated after the slave node executes the operator. The execution information includes the execution sub-information of multiple operators, and each execution sub-information includes the execution time of the operator, memory usage, disk read data volume, network transmission data volume, number of output data rows, etc. Figure 3 As shown, the step of storing the execution information in a predetermined memory of the slave node also includes:

[0055] Step S21: Acquire the collected execution sub-information and determine the identity information of the operator corresponding to the execution sub-information. The identity information of the operator is used to uniquely identify the operator. In the embodiment of the present invention, the second meta-information of the operator can be used as its identity information to identify it. The second meta-information includes the operator ID (ID is the abbreviation of Identity document, which is the abbreviation of identity identification number, account number, unique code, etc.), the parent operator and child operator of the operator, the execution time when the operator starts to execute, the ID of the query task to which the operator belongs, etc.

[0056] Step S22: based on the identity information, the execution sub-information is stored in a hash table, wherein the storage structure of the hash table includes a key Key and a value Value, the Key is the identity information, and the Value is the execution sub-information. That is, the embodiment of the present invention stores the real-time data of the execution sub-information according to the Key-Value of the hash table.

[0057] The embodiment of the present invention stores the execution sub-information of each operator in the query task execution process through a hash table, thereby achieving more detailed recording of the execution status of each operator of the query task, improving the query performance, and collecting more refined data to more accurately diagnose the performance of the data warehouse.

[0058] When the embodiment of the present invention uses a hash table to initially store the execution sub-information of the corresponding operator, the execution sub-information of the operator is initialized in the hash table to obtain the initialization data of the execution sub-information. As the operator is executed, the initialization data is continuously updated to obtain the real-time data of the operator. Figure 4 As shown, the step of storing the execution sub-information in the hash table based on the identity information includes:

[0059] Step S31: Collect update information of execution sub-information. Update information is information generated when the slave node executes the operator to the predetermined execution node. When the predetermined execution node, such as the operator, processes a predetermined amount of data, that is, each time a batch of data is processed, update information of the current operator is collected. The update information is the real-time data of the execution sub-information when the operator executes to the corresponding execution node.

[0060] Step S32: Determine the target Value in the hash table for storing the execution sub-information corresponding to the operator according to the identity information.

[0061] Step S33: Update the target Value based on the update information, that is, update the execution sub-information stored in the target Value, and ensure the real-time and accuracy of the execution sub-information.

[0062] Therefore, as the operator is executed, the embodiment of the present invention queries the target Value corresponding to the operator from the hash table according to the identity information of the operator after collecting the update information of the execution sub-information, thereby updating the execution sub-information stored in the target Value based on the update information, thereby ensuring the real-time and accuracy of the execution sub-information.

[0063] In step S32, the step of updating the execution sub-information stored in the target Value based on the update information includes: after setting a row-level lock at the target Value in the hash table, updating the execution sub-information stored in the target Value based on the update information. The row-level lock only locks the execution sub-information stored in the target Value that needs to be updated, has a small locking granularity, a low probability of lock conflict, and a high concurrency, which can prevent the execution processes of various query tasks in the same slave node from competing for locks and affecting the concurrency performance of the data warehouse.

[0064] like Figure 5 As shown, before determining the performance of the data warehouse according to the processing information in step S12, the method provided by the embodiment of the present invention further includes the following steps:

[0065] Step S41: Pre-aggregate the execution sub-information at the slave node to obtain pre-processing information. The pre-processing here is a sub-computation process of the aggregation process performed by the master node, and the calculation processing of the sub-computation process can be completed in the slave node in advance. For example, when the master node needs to calculate the total execution time of the query task, the execution time of the operator of the slave node can be summed up in the slave node first (i.e., pre-processing).

[0066] Step S42: Send the preprocessing information to the master node, so that the master node aggregates the preprocessing information to obtain the processing information. The master node performs final aggregation processing on the preprocessing information sent from the slave node, such as performing final summation on the preprocessing information obtained in the previous step to obtain the total execution time, thereby reducing the computational burden that the master node has to bear when performing the aggregation processing task.

[0067] The embodiment of the present invention maintains query statistics in a distributed manner. Each server node (mainly slave nodes) of the data warehouse can independently maintain the execution information of all query tasks currently running on the slave node, and evenly distribute the work overhead of collecting and managing the execution information to each slave node, thereby reducing the performance impact of the collection of execution information on the query link itself. By maintaining the execution information of locally executed query tasks on each slave node, the overhead of network transmission can be reduced and the query performance of the data warehouse can be improved. Compared with the traditional centralized information maintenance mode, the embodiment of the present invention has higher concurrency and scalability. Secondly, the load of the aggregation processing of the execution information by the master node is distributed to the data pulling stage of pulling the execution information from the disk or hash table, and the data pulling stage is less sensitive to performance than the execution stage of the query task executed by the slave node.

[0068] In addition, each slave node in the embodiment of the present invention uses a shared reserved memory locally to maintain the real-time status of the query task in operation, that is, the real-time data of the execution information collected under the execution process of different query tasks is uniformly maintained and stored through the reserved memory. Compared with disk storage, the read and write speed of the reserved memory is faster, which can reduce the disk read and write overhead, thereby improving the maintenance performance of the execution information. The reserved memory uses a hash table to efficiently store the first metadata and execution information of the query task. The cost of retrieving the execution sub-information of each operator in the query task is low, and the concurrency is high. When the hash table is used to maintain the real-time data of the execution information, the update overhead of the execution information can be reduced, and the impact on the concurrent performance of the query link when maintaining the execution information can be reduced. Therefore, the embodiment of the present invention can adopt a higher data update frequency under the premise of reducing the overhead as much as possible, which can enhance the real-time and accuracy of the execution information, thereby ensuring the accuracy of the performance diagnosis results of the data warehouse.

[0069] A second embodiment of the present invention provides a performance diagnosis system for a data warehouse, and the performance diagnosis system is used to execute the performance diagnosis method for a data warehouse provided by the first embodiment of the present invention.

[0070] The performance diagnosis system provided by the embodiment of the present invention can adopt a distributed information maintenance mode in the process of executing query tasks in parallel for multiple slave nodes, store the collected execution information of the query tasks on the disk of the corresponding slave node, and maintain the execution information of the query tasks currently executed by the slave node at each slave node. When the execution information stored in the statistics of the slave node is accessed, the execution information read from the disk is sent to the master node, so that the execution information is aggregated at the master node. In this way, the network transmission overhead generated by collecting the execution information is amortized from the information collection stage to the execution information access stage. In this way, after the slave node executes the query task, the diagnosis system only needs to perform a disk write operation on the execution information locally, and does not need to transmit data in the network. Therefore, the network transmission overhead can be greatly reduced, thereby improving the query performance of the data warehouse.

[0071] A third embodiment of the present invention provides a data warehouse, which includes the performance diagnosis system of the data warehouse provided by the first embodiment of the present invention.

[0072] A fourth embodiment of the present invention provides an electronic device, including: a processor, and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the performance diagnosis method of the data warehouse provided by the first embodiment of the present invention.

[0073] A fifth embodiment of the present invention provides a non-transitory machine-readable medium storing computer instructions, where the computer instructions are used to cause a computer to execute the data warehouse performance diagnosis method provided by the first embodiment of the present invention.

[0074] The sixth embodiment of the present invention provides an application embodiment of a performance diagnosis method for executing a data warehouse through a performance diagnosis system on the basis of the above five embodiments. In this application embodiment, the architecture of the data warehouse is an MPP architecture in a master-slave working mode, including a master node and a slave node. The master node is used for distributing computing tasks (such as query tasks), and the slave node is used for storing data and executing computing tasks.

[0075] The core point of the application embodiment of the present invention is to face the query tasks executed in parallel by multiple slave nodes, and accordingly adopt a distributed information maintenance mode to maintain the execution information of all query tasks of the current slave node in each slave node. In this way, after the slave node executes the query task, it only needs to perform a disk write operation locally, and does not need to transmit the execution information in the network, so it can greatly reduce the overhead of network transmission, thereby improving the query performance of the slave node.

[0076] When the execution information is accessed, the performance diagnosis system performs data calculation and network transmission, thereby allocating the overhead of the performance diagnosis system from the execution information collection stage to the access stage. Compared with the query execution task itself, accessing execution information is less sensitive to the performance of the data warehouse.

[0077] At the same time, each slave node of the application embodiment of the present invention uses the reserved memory shared by each execution process to maintain the real-time data of the execution information. In order to collect the real-time data of the query task execution information in real time, it is necessary to maintain and modify the real-time data at a high frequency. Compared with disk storage, the read and write speed of the reserved memory to maintain the execution information is faster, which can reduce the disk read and write overhead of the data warehouse, thereby further improving the query performance of the data warehouse. In addition, since the real-time data of the execution information is directly read and written in the reserved memory without the need for file system operations, it can also alleviate problems such as lock contention caused by concurrent read and write operations of the file system, thereby improving the concurrency performance of each server node of the data warehouse.

[0078] The following is a detailed description of the process of the performance diagnosis system executing the performance diagnosis method:

[0079] Step 1: Figure 2 As shown, when a query task in the data warehouse starts to execute, the first metadata of the query task (query task ID, user, queried warehouse, start time, etc.) will be recorded on the master node, and then the master node will schedule the slave node to execute the query task according to the query execution plan (each slave node will receive the query execution plan sent by the master node, and then execute the query task according to this plan).

[0080] Step 2: After receiving the information from the master node that the corresponding query task needs to be executed, the slave node records the first metadata of the query task in a shared hash table in a predetermined memory. Each execution process of the slave node can retrieve the complete information of the query task from the shared hash table.

[0081] Step 3, starting from the executor of the node, the operators of this query task are executed in order. When each operator starts to execute, the performance diagnosis system registers the operator's second metadata (operator ID, parent operator and child operator of the operator, execution time when the operator starts to execute, and ID of the query task to which the operator belongs) into the hash table used to store the operator's execution sub-information (the hash table here and the above-mentioned shared hash table are different hash tables located in the predetermined memory), and initializes the operator's execution sub-information (including execution time, memory usage, disk read data volume, network transmission data volume, number of output data rows, etc.). The application embodiment of the present invention can regard this hash table as a data table that stores the operator's execution sub-information according to the Key-Value data structure, wherein the Key is the operator's second metadata, the hash table uniquely locates an operator through the second metadata, and the Value is the execution sub-information collected when the operator is executed.

[0082] Step 4: During the process of executing the operator from the node, the operator queries the data entry corresponding to the operator from the hash table according to the first metadata each time it processes a batch of data (i.e., finds the target Value corresponding to the operator), and updates the data of the execution information in the target Value once. It should be noted that when the performance diagnosis system in this application embodiment updates the execution sub-information of the operator stored in the hash table, it only needs to add a row-level lock to the hash table, without adding a larger range lock to the entire hash table, so as to avoid the query tasks in the same slave node competing for the lock and affecting the concurrent performance of the data warehouse.

[0083] Step 5: Figure 6 As shown in the figure, after the slave node executes a predetermined operator (such as data scanning or data management, etc.), the performance diagnosis system takes out the target Value corresponding to the predetermined operator from the hash table, and combines the second metadata and execution sub-information of the operator itself and writes it to the disk of the slave node. When the user accesses it later, the corresponding execution sub-information can be read from the disk based on the second metadata. At the same time, the execution sub-information of the operator is deleted from the hash table, which improves the storage performance of the hash table for the execution sub-information of other operators and ensures the real-time and accuracy of other execution sub-information.

[0084] After the application embodiment of the present invention collects the execution information of the query task based on the above five steps, the performance diagnosis system does not perform aggregation processing (also known as aggregation operation) on the execution information, but only stores the original execution information on the disk of the corresponding slave node. The aggregation operation is performed through the master node when the user requests to access the execution information of the query task.

[0085] Specifically, when the user accesses the execution information, the performance diagnosis system will transmit the execution information (specifically, the execution sub-information of each operator) read from each disk by each slave node to the master node. Before transmitting the execution information to the master node, the slave node will first pre-aggregate the execution information to a certain extent, thereby reducing the burden of the master node's computing tasks, and then transmit the pre-aggregated pre-processed information to the master node, and the master node will complete the final aggregation task. For example, to obtain the total execution time of a query task, the master node is required to sum the execution time of all operators. For this, the application embodiment of the present invention will first sum the execution time of the operators of this node locally in the slave node (i.e., pre-processing). After the summation result is transmitted back to the master node, the master node will make the final summation based on the summation results uploaded by each slave node. The performance diagnosis system supports access to the real-time data of the query task and the execution information of the historical query task. When the user needs to access the real-time data of the query task, the performance diagnosis system will read the real-time data from the hash table of the predetermined memory on the node. When accessing the execution information of historical query tasks, the performance diagnosis system obtains the execution information of historical query tasks by reading the data files on the disk.

[0086] The embodiment of the present invention maintains query statistics in a distributed manner. Each server node (mainly slave nodes) of the data warehouse can independently maintain the execution information of all query tasks currently running on the slave node, and evenly distribute the work overhead of collecting and managing the execution information to each slave node, thereby reducing the performance impact of the collection of execution information on the query link itself. By maintaining the execution information of locally executed query tasks on each slave node, the overhead of network transmission can be reduced and the query performance of the data warehouse can be improved. Compared with the traditional centralized information maintenance mode, the embodiment of the present invention has higher concurrency and scalability. Secondly, the load of the aggregation processing of the execution information by the master node is distributed to the data pulling stage of pulling the execution information from the disk or hash table, and the data pulling stage is less sensitive to performance than the execution stage of the query task executed by the slave node.

[0087] In addition, each slave node in the embodiment of the present invention uses a shared reserved memory locally to maintain the real-time status of the query task in operation, that is, the real-time data of the execution information collected under the execution process of different query tasks is uniformly maintained and stored through the reserved memory. Compared with disk storage, the read and write speed of the reserved memory is faster, which can reduce the disk read and write overhead, thereby improving the maintenance performance of the execution information. The reserved memory uses a hash table to efficiently store the first metadata and execution information of the query task. The cost of retrieving the execution sub-information of each operator in the query task is low, and the concurrency is high. When the hash table is used to maintain the real-time data of the execution information, the update overhead of the execution information can be reduced, and the impact on the concurrent performance of the query link when maintaining the execution information can be reduced. Therefore, the embodiment of the present invention can adopt a higher data update frequency under the premise of reducing the overhead as much as possible, which can enhance the real-time and accuracy of the execution information, thereby ensuring the accuracy of the performance diagnosis results of the data warehouse.

[0088] The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of an embodiment of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0091] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0092] The various steps described in the method implementation methods provided in the embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0093] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0094] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A performance diagnosis method for a data warehouse, characterized in that: The data warehouse includes a master node and a slave node, and the method includes: The collected execution information is stored in the disk of the slave node, wherein the execution information is information generated during the process of the slave node executing the query task of the data warehouse; In response to an access request for the execution information, the execution information read from the disk is sent to the master node, so that the execution information is aggregated at the master node to obtain processing information, and the performance of the data warehouse is determined based on the processing information.

2. The method according to claim 1, characterized in that: Before the master node aggregates the execution information to obtain processing information, the method further includes: The execution information is stored in a predetermined memory of the slave node, wherein the predetermined memory is a memory shared by execution processes executing different query tasks on the slave node; Then, before the master node aggregates the execution information to obtain processing information, the method further includes: In response to an access request for the execution information, the execution information read from the predetermined memory is sent to the master node.

3. The method according to claim 2, characterized in that The step of storing the execution information in a predetermined memory of the slave node comprises: Creating a hash table in the predetermined memory to store the execution information in the hash table; Then, the step of storing the collected execution information in the disk of the slave node includes: The execution information read from the hash table is written to the disk.

4. The method according to claim 2, characterized in that: A hash table is created in the predetermined memory, the query task includes multiple operators, and after the slave node executes the operator, execution sub-information is generated, the execution information includes the execution sub-information of the multiple operators. The step of storing the execution information in the predetermined memory of the slave node includes: Acquire the collected execution sub-information, and determine the identity information of the operator corresponding to the execution sub-information; The execution sub-information is stored in the hash table based on the identity information, wherein the storage structure of the hash table includes a key Key and a value Value, the Key is the identity information, and the Value is the execution sub-information.

5. The method according to claim 4, characterized in that The step of storing the execution sub-information in the hash table based on the identity information comprises: Collecting update information of the execution sub-information, where the update information is information generated when the slave node executes the operator to a predetermined execution node; Determine, according to the identity information, a target Value in the hash table for storing the execution sub-information corresponding to the operator; The target Value is updated based on the update information.

6. The method according to claim 5, characterized in that The step of updating the execution sub-information stored in the target Value based on the update information includes: After setting a row-level lock at the target Value of the hash table, the execution sub-information stored in the target Value is updated based on the update information.

7. The method according to claim 4, characterized in that Before determining the performance of the data warehouse according to the processing information, the method further includes: Pre-aggregating the execution sub-information at the slave node to obtain pre-processing information; The pre-processing information is sent to the master node, so that the master node aggregates the pre-processing information to obtain the processing information.

8. A performance diagnosis system for a data warehouse, characterized in that: The diagnostic system is used to execute the performance diagnostic method described in any one of claims 1 to 7.

9. A data warehouse, characterized in that: The data warehouse includes the performance diagnosis system described in claim 8.

10. An electronic device, comprising: A processor and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

11. A non-transitory machine-readable medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images