Information generating method, device and device for generating information

Abstract

The embodiment of the present application discloses an information generation method, device and device for generating information. The embodiment of the method includes: receiving an SQL statement for a data warehouse and determining the task corresponding to the SQL statement; obtaining the execution plan information of the SQL statement and the image information of the task; summarizing the execution plan information and the image information to generate the execution plan auxiliary information of the SQL statement. This implementation method facilitates users to understand the execution plan and related information of the SQL statement, and improves the richness and readability of the information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an information generating method, device and a device for generating information. Background Art

[0002] Hive is a data warehouse tool based on the distributed system infrastructure Hadoop, which is used to extract, transform and load data. Hive can map structured data files to a database table, and provide SQL (Structured Query Language) query functions, which can convert SQL statements into MapReduce (a programming model) tasks for execution. Usually, users can understand the task execution process through the Hive execution plan.

[0003] In the prior art, the execution plan provided by Hive is usually used directly and returned to the user. However, the execution plan provided by Hive is usually poorly readable, and users who are not familiar with the underlying technology of MapReduce find it difficult to understand its meaning. At the same time, the execution plan provided by Hive contains limited information and cannot provide users with more sufficient information. Summary of the invention

[0004] The embodiments of the present application propose an information generation method, device, and device for generating information, so as to facilitate users to understand the execution plan and related information of SQL statements and improve the richness and readability of the information.

[0005] In a first aspect, an embodiment of the present application provides an information generation method, the method comprising: receiving an SQL statement for a data warehouse, and determining a task corresponding to the SQL statement; obtaining execution plan information of the SQL statement and portrait information of the task; aggregating the execution plan information and the portrait information to generate execution plan auxiliary information of the SQL statement.

[0006] In the second aspect, an embodiment of the present application provides an information generating device, which includes: a determination unit, configured to receive an SQL statement for a data warehouse and determine a task corresponding to the SQL statement; an acquisition unit, configured to obtain execution plan information of the SQL statement and portrait information of the task; and a generation unit, configured to summarize the execution plan information and the portrait information to generate execution plan auxiliary information of the SQL statement.

[0007] In a third aspect, an embodiment of the present application provides a device for generating information, comprising a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: receiving an SQL statement for a data warehouse, and determining a task corresponding to the SQL statement; obtaining execution plan information of the SQL statement and portrait information of the task; and summarizing the execution plan information and the portrait information to generate execution plan auxiliary information of the SQL statement.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0009] The information generation method, device and device for generating information provided in the embodiments of the present application receive SQL statements for a data warehouse and determine the tasks corresponding to the SQL statements; then obtain the execution plan information of the SQL statements and the image information of the tasks; finally, the execution plan information and the image information are summarized to generate the execution plan auxiliary information of the SQL statements. Since the execution plan auxiliary information includes not only the execution plan information but also the image information of the tasks, it can not only assist users in understanding the execution plan and increase the readability of the execution plan, but also improve the richness of the information. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is a flow chart of an embodiment of the information generation method according to the present application;

[0012] Figure 2 is a schematic diagram of an execution plan auxiliary diagram according to the present application;

[0013] Figure 3 is a schematic structural diagram of an embodiment of an information generating device according to the present application;

[0014] Figure 4 is a schematic diagram of the structure of a device for generating information according to the present application;

[0015] Figure 5 It is a schematic diagram of the structure of the server in some embodiments of the present application. DETAILED DESCRIPTION

[0016] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] Please refer to Figure 1 , which shows a process 100 of an embodiment of the information generation method according to the present application. The above information generation method can be run on various electronic devices, including but not limited to: servers, smart phones, tablet computers, laptop computers, car computers, desktop computers, etc.

[0019] The information generation method in this embodiment may include the following steps:

[0020] Step 101, receiving an SQL statement for a data warehouse, and determining a task corresponding to the SQL statement.

[0021] In this embodiment, the execution subject of the information generation method (such as the above-mentioned electronic device) can receive the SQL statement for the data warehouse sent by the user, and determine the task corresponding to the SQL statement. Among them, the data warehouse can be Hive. The SQL statement for the data warehouse is the Hive SQL statement. The task corresponding to the SQL statement is the MapReduce (a programming model) task (Job) converted during the execution of the SQL statement by Hive. Hive needs to convert the SQL statement into a MapReduce task for execution when working. Therefore, after receiving the Hive SQL statement, the above-mentioned execution subject can determine the MapReduce task converted by Hive, thereby obtaining the task corresponding to the SQL statement. Here, the number of tasks can be one or more.

[0022] In practice, Hive is a data warehouse tool based on the distributed system infrastructure Hadoop, which is used for data extraction, transformation, and loading. Hive can map structured data files into a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution. Among them, MapReduce is a programming model for parallel computing of large-scale data sets. The main idea of ​​MapReduce is to specify a Map function to map a set of key-value pairs into a new set of key-value pairs, and to specify a concurrent Reduce function to ensure that each of all mapped key-value pairs shares the same key group. Hive provides a rich set of Hive SQL query methods to analyze data stored in the Hadoop distributed file system, so that users who are not familiar with MapReduce can easily use the Hive SQL language to query, summarize and analyze data, providing good scalability and extensibility for big data data operations.

[0023] When Hive executes SQL statements, it will first parse the SQL statements into an Abstract Syntax Tree (AST), or syntax tree for short. An abstract syntax tree is an abstract representation of the syntax structure of the source code. It represents the syntax structure of the programming language in a tree-like form, and each node on the tree represents a structure in the source code.

[0024] After being parsed into an abstract syntax tree, the complexity of the abstract syntax tree is still high and it is not easy to translate into a MapReduce task. Therefore, it needs to be further abstracted and structured to form the basic component unit of the query (QueryBlock, QB). QB is the most basic component unit of a SQL statement, which includes three parts: input source, calculation process, and output. It can be understood that a QB is a subquery. Hive will traverse the abstract syntax tree and perform syntax parsing to obtain QB. In this process, you can verify the table name and column name in the SQL statement, check and implicitly convert data types, and verify the functions provided by Hive and user-defined functions (User Defined Function, UDF).

[0025] After the abstract syntax tree is abstracted into QB, logical plan generation and logical plan optimization will be carried out in sequence. Among them, logical plan generation is to generate a logical plan, that is, to generate an operator tree (Operator Tree), also known as an operation tree, by traversing QB. Logical plan optimization is to optimize the operator tree, including column pruning, partition pruning, etc., to merge operators to achieve the purpose of reducing Map Reduce tasks.

[0026] After the logical plan optimization, Hive will generate a physical plan. Physical plan generation refers to traversing the operator tree and converting the operator tree into a MapReduce task. Then, the MapReduce task can be optimized by the physical layer optimizer to generate the final execution plan. The task corresponding to the SQL statement in this embodiment is the MapReduce task in the final execution plan. In practice, the execution plan can be sent to the Hadoop cluster for execution, thereby obtaining the execution result of the SQL statement.

[0027] Step 102, obtaining the execution plan information of the SQL statement and the profile information of the task.

[0028] In this embodiment, the execution plan information may be the execution plan information obtained by successively parsing the SQL statement into an abstract syntax tree, converting the abstract syntax tree into basic components of a query, traversing the basic components of the query to generate an operator tree, optimizing the operator tree, converting the optimized operator tree into a MapReduce task, and optimizing it. The execution plan information may include the specific operations of each task and the execution order of the tasks, etc.

[0029] The task profile information can be obtained by parsing the target path (such as the Yarn JobHistory path) through the MapReduce program. The above-mentioned target path (such as the Yarn JobHistory path) is used to store the task history record, which may include task-related data, such as the task profile information and other data. Here, the task profile information may include various information related to the task. For example, it may include but is not limited to the data table information operated by the task, the status information of the task, the task submission time, the task start time, the task completion time, the number of mapping tasks (Mapper Task), the number of reduce tasks (Reducer Task), the number of bytes read, the number of bytes written, etc.

[0030] In some optional implementations of this embodiment, the execution subject may generate the execution plan information of the SQL statement according to the following steps:

[0031] The first step is to extract the optimization parameters and user-defined functions of the above SQL statements from the historical scheduling information of the above tasks.

[0032] Specifically, the scheduling log can be obtained through the log interface provided by the scheduling system. Then, the historical scheduling information of the above task can be obtained from the scheduling log. The above scheduling system can be a system for scheduling Hive SQL, and the scheduling system can schedule Hive SQL on a daily basis.

[0033] The historical scheduling information may include but is not limited to the optimization parameters of the above SQL statements and user-defined functions. In practice, Hive comes with some functions, but since the number of built-in functions is limited, users can extend it by defining custom functions. User-defined functions can affect the execution plan based on user settings.

[0034] Optimization parameters can be used to optimize Hive. For example, when there are no optimization parameters, Hive executes tasks in series. When an optimization parameter is added, tasks can be executed in parallel. Therefore, the optimization parameters can also change the execution plan from serial to parallel.

[0035] The second step is to obtain the syntax tree corresponding to the SQL instruction. The syntax tree is the above-mentioned abstract syntax tree, which will not be described here.

[0036] The third step is to generate the execution plan information of the SQL statement based on the optimization parameters, the user-defined function and the syntax tree. Here, the syntax tree can be first converted into a MapReduce task, and then the MapReduce task can be optimized based on the user-defined function and the optimization parameters to obtain the execution plan information of the SQL statement. The process of converting the syntax tree into a MapReduce task can be referred to the above description and will not be repeated here.

[0037] In some optional implementations of this embodiment, the execution subject may obtain the task portrait information through the following steps:

[0038] The first step is to obtain the initial profile information of the task from the target path (such as the Yarn JobHistory path).

[0039] Optionally, the above initial portrait information may include but is not limited to at least one of the following: data table information operated by the task, task status information, task submission time, task start time, task completion time, number of mapping tasks, number of reduction tasks, number of bytes read and number of bytes written, etc.

[0040] The second step is to obtain the diagnostic information of the task, which is used to characterize the health of the task execution results.

[0041] Here, the diagnostic information of the task is obtained through the analysis of the Dr.elephant tool. The Dr.elephant tool is a tool for performance monitoring and tuning of Hadoop and Spark (a fast and general computing engine designed for large-scale data processing) tasks. It can automatically collect all metrics of the Hadoop platform, analyze the collected data, and display the analysis results in a simple and easy-to-understand form. The Dr.elephant tool is designed to guide Hadoop or Spark developers to conveniently optimize their tasks through its analysis results, thereby improving the efficiency of developers and the efficiency of using Hadoop clusters.

[0042] Optionally, the above diagnostic information may include but is not limited to at least one of the following: the severity status of the task, whether the task has data skew, whether the resource utilization of the task is reasonable, and other health information. Among them, data skew is very common in the MapReduce programming model. Data skew means that a large number of identical keys are assigned to a partition, causing a node to bear huge pressure, and other nodes have to wait for this busy node after the calculation is completed, thus dragging down the overall calculation time and resulting in low processing efficiency.

[0043] Step 103, summarize the execution plan information and the profile information to generate execution plan auxiliary information of the SQL statement.

[0044] In this embodiment, the execution subject may summarize the execution plan information and the portrait information to generate the execution plan auxiliary information of the SQL statement. The execution plan auxiliary information here may be information that summarizes the execution plan information and the portrait information, and the information may be in various forms, such as text, image, graphic, etc. The information may include the specific operations of each task and the execution order of the tasks.

[0045] In some optional implementations of this embodiment, the execution subject may draw an auxiliary graph of the execution plan of the SQL statement through a preset drawing tool, and the auxiliary graph of the execution plan includes the execution plan information and the portrait information. For example, an AGraph class in the pygraphviz module in the python programming language may be used to create an instance of a graph, and then some methods may be called with the instance to construct points and establish edges to characterize the order and relationship of tasks, thereby obtaining an auxiliary graph of the execution plan.

[0046] As an example, Figure 2The schematic diagram of the execution plan auxiliary graph is shown. The execution plan auxiliary graph shown includes the Job (task) id (identification), related tables, Job health, whether the data is skewed, Job submission time, Job start time, Job completion time, Job running time, Map (mapping) read-in bytes, Reduce (reduction) read-in bytes, Map write bytes, Reduce write bytes, etc. In addition, it can also include condition selectors, specific operations of each stage (Stage) and other information, such as move operations, merge file operations, etc.

[0047] It should be noted that Figure 2 Only one task is shown, but the present application is not limited to the case of only one task. When a SQL statement has multiple tasks, the execution plan and profile information of each task can be summarized into an execution plan auxiliary graph, and the order of tasks and other information can be shown in the execution plan auxiliary graph.

[0048] In some optional implementations of this embodiment, after generating the auxiliary information of the execution plan of the SQL statement, the execution subject may also take the user who sent the SQL statement as the target user, and in response to receiving the information execution plan auxiliary information acquisition request sent by the target user, return the auxiliary information of the execution plan to the target user. After the user obtains the model, he can effectively view the multi-dimensional information such as the profile information and health information of the task corresponding to the user's Hive SQL.

[0049] The method provided by the above embodiment of the present application receives an SQL statement for a data warehouse and determines the task corresponding to the SQL statement; then obtains the execution plan information of the SQL statement and the image information of the task; and finally summarizes the execution plan information and the image information to generate the execution plan auxiliary information of the SQL statement. Thus, the execution plan auxiliary information not only includes the execution plan information, but also includes the image information of the task, which can not only assist the user in understanding the execution plan and increase the readability of the execution plan, but also improve the richness of the information.

[0050] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an information generating device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0051] like Figure 3As shown, the above-mentioned information generating device 300 of this embodiment includes: a determination unit 301, configured to receive an SQL statement for a data warehouse, and determine a task corresponding to the above-mentioned SQL statement; an acquisition unit 302, configured to acquire execution plan information of the above-mentioned SQL statement and portrait information of the above-mentioned task; a generation unit 303, configured to summarize the above-mentioned execution plan information and the above-mentioned portrait information, and generate execution plan auxiliary information of the above-mentioned SQL statement.

[0052] In some optional implementations of the present embodiment, the acquisition unit 302 is further configured to acquire the execution plan information through the following steps: extracting the optimization parameters and user-defined functions of the SQL statement from the historical scheduling information of the task; acquiring the syntax tree corresponding to the SQL instruction; and generating the execution plan information of the SQL statement based on the optimization parameters, the user-defined function and the syntax tree.

[0053] In some optional implementations of the present embodiment, the acquisition unit 302 is further configured to acquire the portrait information through the following steps: acquiring the initial portrait information of the task from the target path; acquiring the diagnostic information of the task, the diagnostic information being used to characterize the health of the execution result of the task; and aggregating the initial portrait information and the diagnostic information into target portrait information.

[0054] In some optional implementations of this embodiment, the above-mentioned initial portrait information includes at least one of the following: data table information operated by the task, task status information, task submission time, task start time, task completion time, number of mapping tasks, number of reduction tasks, number of bytes read and number of bytes written.

[0055] In some optional implementations of this embodiment, the above-mentioned diagnostic information includes at least one of the following: health information such as the severity status of the task, whether the task has data skew, and whether the resource utilization of the task is reasonable.

[0056] In some optional implementations of this embodiment, the above-mentioned generation unit 303 is further configured to: draw an execution plan auxiliary diagram of the above-mentioned SQL statement through a preset drawing tool, and the above-mentioned execution plan auxiliary diagram includes the above-mentioned execution plan information and the above-mentioned portrait information.

[0057] In some optional implementations of this embodiment, the above-mentioned device also includes: a sending unit, which is configured to take the user who sends the above-mentioned SQL statement as the target user, and return the above-mentioned execution plan auxiliary information to the above-mentioned target user in response to receiving an information execution plan auxiliary information acquisition request sent by the above-mentioned target user.

[0058] The device provided by the above embodiment of the present application receives the SQL statement for the data warehouse and determines the task corresponding to the SQL statement; then obtains the execution plan information of the SQL statement and the image information of the task; finally, the execution plan information and the image information are summarized to generate the execution plan auxiliary information of the SQL statement. Therefore, the execution plan auxiliary information not only includes the execution plan information, but also includes the image information of the task, which can not only assist the user to understand the execution plan and increase the readability of the execution plan, but also improve the richness of the information.

[0059] Figure 4 4 is a block diagram of an input device 400 according to an exemplary embodiment, and the device 400 may be a smart terminal or a server. For example, the device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0060] Reference Figure 4 , the device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0061] The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-described method. In addition, the processing component 402 may include one or more modules to facilitate interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate interaction between the multimedia component 408 and the processing component 402.

[0062] The memory 404 is configured to store various types of data to support operations on the device 400. Examples of such data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0063] The power supply component 406 provides power to the various components of the device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 400.

[0064] The multimedia component 408 includes a screen that provides an output interface between the above-mentioned device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The above-mentioned touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the above-mentioned touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0065] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the device 400 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0066] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0067] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the device 400. For example, the sensor assembly 414 can detect the open / closed state of the device 400, the relative positioning of the components, such as the display and keypad of the device 400, the sensor assembly 414 can also detect the position change of the device 400 or a component of the device 400, the presence or absence of user contact with the device 400, the orientation or acceleration / deceleration of the device 400 and the temperature change of the device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0068] The communication component 416 is configured to facilitate wired or wireless communication between the device 400 and other devices. The device 400 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0069] In an exemplary embodiment, the apparatus 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0070] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of the device 400 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0071] Figure 55 is a schematic diagram of the structure of the server in some embodiments of the present application. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 522 (for example, one or more processors) and a memory 532, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 542 or data 544. Among them, the memory 532 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 522 can be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the server 500.

[0072] The server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, one or more keyboards 556, and / or one or more operating systems 541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0073] A non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a device (smart terminal or server), the device is able to execute an information generation method. The method includes: receiving an SQL statement for a data warehouse and determining a task corresponding to the SQL statement; obtaining execution plan information of the SQL statement and portrait information of the task; and summarizing the execution plan information and the portrait information to generate execution plan auxiliary information of the SQL statement.

[0074] Optionally, the execution plan information is obtained through the following steps: extracting the optimization parameters and user-defined functions of the SQL statement from the historical scheduling information of the task; obtaining the syntax tree corresponding to the SQL instruction; and generating the execution plan information of the SQL statement based on the optimization parameters, the user-defined function and the syntax tree.

[0075] Optionally, the portrait information is obtained through the following steps: obtaining initial portrait information of the task from the target path; obtaining diagnostic information of the task, wherein the diagnostic information is used to characterize the health of the execution result of the task; and aggregating the initial portrait information and the diagnostic information into target portrait information.

[0076] Optionally, the initial portrait information includes at least one of the following: data table information operated by the task, task status information, task submission time, task start time, task completion time, number of mapping tasks, number of reduction tasks, number of bytes read, and number of bytes written.

[0077] Optionally, the diagnostic information includes at least one of the following: health information such as the severity status of the task, whether the task has data skew, and whether the resource utilization of the task is reasonable.

[0078] Optionally, the execution plan information and the portrait information are summarized to generate auxiliary information of the execution plan of the SQL statement, including: drawing an auxiliary graph of the execution plan of the SQL statement through a preset drawing tool, wherein the auxiliary graph of the execution plan includes the execution plan information and the portrait information.

[0079] Optionally, the device is configured to execute the one or more programs including instructions for performing the following operations by one or more processors: taking the user sending the SQL statement as the target user, and returning the execution plan auxiliary information to the target user in response to receiving a request for obtaining execution plan auxiliary information sent by the target user.

[0080] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0081] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0082] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0083] The above is a detailed introduction to an information generating method, a device and a device for generating information provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating information, characterized in that: The method comprises: Receive an SQL statement for a data warehouse and determine a task corresponding to the SQL statement; Obtain the execution plan information of the SQL statement and the portrait information of the task; the execution plan information is obtained by the following steps: extracting the optimization parameters and user-defined functions of the SQL statement from the historical scheduling information of the task; obtaining the syntax tree corresponding to the SQL instruction; obtaining the basic components of the query according to the syntax tree, and generating an operator tree based on the basic components of the query; generating the execution plan information of the SQL statement based on the optimization parameters, the user-defined function and the syntax tree, including: generating a MapReduce task according to the operator tree, and obtaining the execution plan information based on the user-defined function, the optimization parameters and the MapReduce task; the portrait information is obtained by the following steps: obtaining the initial portrait information of the task from the target path; obtaining the diagnostic information of the task, the diagnostic information being used to characterize the health of the execution result of the task; and summarizing the initial portrait information and the diagnostic information into target portrait information; The execution plan information and the profile information are aggregated to generate execution plan auxiliary information of the SQL statement.

2. The method according to claim 1, characterized in that The initial portrait information includes at least one of the following: The data table information operated by the task, task status information, task submission time, task start time, task completion time, number of mapping tasks, number of reduction tasks, number of bytes read, and number of bytes written.

3. The method according to claim 1, characterized in that The diagnostic information includes at least one of the following health information: The severity status of the task, whether the task has data skew, and whether the resource utilization of the task is reasonable.

4. The method according to claim 1, characterized in that The step of aggregating the execution plan information and the profile information to generate auxiliary execution plan information of the SQL statement includes: An auxiliary graph of the execution plan of the SQL statement is drawn by a preset drawing tool, wherein the auxiliary graph of the execution plan includes the execution plan information and the portrait information.

5. The method according to claim 1, characterized in that After generating the execution plan auxiliary information of the SQL statement, the method further includes: The user who sends the SQL statement is regarded as a target user, and in response to receiving a request for obtaining auxiliary information of the execution plan sent by the target user, the auxiliary information of the execution plan is returned to the target user.

6. An information generating device, characterized in that: The device comprises: A determination unit, configured to receive an SQL statement for a data warehouse and determine a task corresponding to the SQL statement; An acquisition unit is configured to acquire the execution plan information of the SQL statement and the portrait information of the task; the execution plan information is acquired through the following steps: extracting the optimization parameters and user-defined functions of the SQL statement from the historical scheduling information of the task; acquiring the syntax tree corresponding to the SQL instruction; obtaining the basic components of the query according to the syntax tree, and generating an operator tree based on the basic components of the query; generating the execution plan information of the SQL statement based on the optimization parameters, the user-defined function and the syntax tree, including: generating a MapReduce task according to the operator tree, and obtaining the execution plan information based on the user-defined function, the optimization parameters and the MapReduce task; the portrait information is acquired through the following steps: acquiring the initial portrait information of the task from the target path; acquiring the diagnostic information of the task, the diagnostic information being used to characterize the health of the execution result of the task; and aggregating the initial portrait information and the diagnostic information into the target portrait information; The generating unit is configured to aggregate the execution plan information and the profile information to generate execution plan auxiliary information of the SQL statement.

7. A device for generating information, characterized in that The system comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations: Receive an SQL statement for a data warehouse and determine a task corresponding to the SQL statement; Acquire the execution plan information of the SQL statement and the profile information of the task; the execution plan information is acquired by the following steps: extract the optimization parameters and user-defined functions of the SQL statement from the historical scheduling information of the task; acquire the syntax tree corresponding to the SQL instruction; obtain the basic component unit of the query according to the syntax tree, and generate the operator tree based on the basic component unit of the query; Based on the optimization parameters, the user-defined function and the syntax tree, the execution plan information of the SQL statement is generated, which includes: generating a MapReduce task according to the operator tree, and obtaining the execution plan information based on the user-defined function, the optimization parameters and the MapReduce task; the portrait information is obtained by the following steps: obtaining the initial portrait information of the task from the target path; obtaining the diagnostic information of the task, the diagnostic information is used to characterize the health of the execution result of the task; and aggregating the initial portrait information and the diagnostic information into the target portrait information; The execution plan information and the profile information are aggregated to generate execution plan auxiliary information of the SQL statement.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images