Big data platform operation and maintenance method, device, computing equipment and storage medium

Abstract

The present invention discloses a big data platform operation and maintenance method, device, computing equipment and storage medium. The method comprises: collecting Hive logs and Yarn logs; parsing the Hive logs to obtain Hive execution information, and writing the Hive execution information into a monitoring table; parsing the Yarn logs to obtain task execution information, and writing the task execution information into the monitoring table; analyzing and counting the Hive execution information to obtain SQL analysis and statistical results; outputting the monitoring table and the SQL analysis and statistical results to a front-end page for display. The present invention collects and analyzes the operation data through the big data platform, and outputs the data to the front-end for display, so as to assist operation and maintenance personnel in performing SQL detection and optimization suggestions, thereby realizing efficient and intelligent operation and maintenance of the big data platform and solving the technical problem that existing operation and maintenance monitoring software and operation and maintenance monitoring methods cannot support daily optimization.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis technology, and in particular to a big data platform operation and maintenance method, apparatus, computing equipment and storage medium. Background Art

[0002] The Hadoop ecosystem (or pan-ecosystem) was born to handle data processing beyond the scale of a single machine. With the popularization of Structured Query Language (SQL) on Hadoop, monitoring and optimizing SQL operations have become a key task in the operation and maintenance of big data platforms, and are fundamental to ensuring the healthy operation of the platforms.

[0003] Existing technology uses CM (Cloudera Manager), a commercial big data platform operations and maintenance software, to monitor SQL execution. However, CM tools can only perform basic operations such as component deployment and process startup and shutdown, and are unable to deeply optimize application access or monitor application resource consumption. Furthermore, SQL execution times can only be manually analyzed from underlying logs to obtain certain system execution information. Once program issues are identified, developers are notified to optimize or modify the application, thereby gradually improving the existing system's execution performance. Existing analysis methods for SQL performance issues generally involve discovering slow SQL execution. Professional maintenance personnel log in to the system and analyze the logs. Based on their professional experience, they analyze the cause of the slowdown and then notify business developers to optimize the system. After the business developers optimize the system, the system is put into operation to observe whether the performance has improved.

[0004] However, when more than 10,000 SQL statements are executed daily, supporting hundreds of accounts, existing operations and maintenance monitoring software and methods become incapable of supporting daily optimization. System performance may be slowed by inefficient SQL statements. Due to the inherently separate design of the Hadoop software architecture, support for operations and maintenance monitoring is significantly weak. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a big data platform operation and maintenance method, apparatus, computing device and storage medium that overcome the above problems or at least partially solve the above problems.

[0006] According to one aspect of the present invention, a big data platform operation and maintenance method is provided, comprising:

[0007] Collect Hive logs and YARN logs;

[0008] Parse Hive logs to obtain Hive execution information, and write the Hive execution information into the monitoring table;

[0009] Parse the Yarn log to obtain task execution information, and write the task execution information into the monitoring table;

[0010] Analyze and count the Hive execution information to obtain SQL analysis and statistical results;

[0011] The monitoring table and the SQL analysis statistical results are output to the front-end page for display.

[0012] According to another aspect of the present invention, a big data platform operation and maintenance device is provided, comprising:

[0013] Log collection module, used to collect Hive logs and Yarn logs;

[0014] A Hive log parsing module is used to parse Hive logs, obtain Hive execution information, and write the Hive execution information into the monitoring table;

[0015] A Yarn log parsing module is used to parse Yarn logs, obtain task execution information, and write the task execution information into the monitoring table;

[0016] An analysis module is used to analyze and count the Hive execution information to obtain SQL analysis and statistical results;

[0017] The output module is used to output the monitoring table and the SQL analysis and statistical results to the front-end page for display.

[0018] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0019] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned big data platform operation and maintenance method.

[0020] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned big data platform operation and maintenance method.

[0021] According to the big data platform operation and maintenance method, apparatus, computing device and storage medium of the present invention, Hive logs and Yarn logs are collected; Hive logs are parsed to obtain Hive execution information, and the Hive execution information is written into a monitoring table; Yarn logs are parsed to obtain task execution information, and the task execution information is written into a monitoring table; Hive execution information is analyzed and counted to obtain SQL analysis and statistical results; the monitoring table and SQL analysis and statistical results are output to a front-end page for display; the present invention collects and analyzes operation data through a big data platform, and then outputs the data to the front-end for display, assisting operation and maintenance personnel in performing SQL detection and optimization suggestions, thereby realizing efficient and intelligent operation and maintenance of the big data platform, and solving the technical problem that existing operation and maintenance monitoring software and operation and maintenance monitoring methods cannot support daily optimization and system performance may be slow due to some inefficient SQL.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0024] Figure 1 The following is a schematic diagram of the Hadoop platform program running system architecture provided by the present invention;

[0025] Figure 2 A flowchart of a big data platform operation and maintenance method provided by an embodiment of the present invention is shown;

[0026] Figure 3 A schematic diagram of the structure of a big data platform operation and maintenance device provided by an embodiment of the present invention is shown;

[0027] Figure 4 A schematic structural diagram of a computing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0029] In the present invention, Figure 1 This is a diagram of the Hadoop platform program running system architecture, such as Figure 1 As shown in the figure, Hadoop is a distributed software processing framework, and Hive is a data warehouse with query capabilities. The Hadoop Distributed File System (HDFS), the underlying Hadoop layer, provides data storage for Hive. By writing SQL commands, developers can access the massive amounts of data stored in Hadoop. Hive's SQL query commands allow developers to bypass MapReduce to a certain extent. Hive can convert SQL scripts into MapReduce tasks to query and analyze HDFS data. The Hadoop platform's implementation includes Hive, Beeline, Pig, and MapReduce. Yarn is a distributed cluster resource management framework. Beeline and Pig are clients for Hadoop data operations and are a data analysis engine that uses specific syntax to manipulate data in HDFS. Yarn, on the other hand, is at a different level from the aforementioned implementations of Hive, Beeline, Pig, and MapReduce. Hive logs reside at the upper level, while Yarn logs reside at the lower level. Since Hive logs are read-only, Yarn services are generally called from the Hive layer. MapReduce is a programming model for parallel computing on large datasets (greater than 1TB). It uses the concepts of "Map" and "Reduce." A Map function is specified to map a set of key-value pairs into a new set of key-value pairs, and a concurrent Reduce function is specified to ensure that all mapped key-value pairs share the same key set. Map represents a calculation performed on each element in a list, while Reduce represents an iterative calculation performed on each element in a list. The specific calculations are performed using the functions passed in, with Map and Reduce providing the computational framework. The Hive layer parses user-submitted SQL into MapReduce tasks for Hadoop to directly run and make data decisions.

[0030] With the popularization of SQL on Hadoop technology, applications developed independently for MapReduce have gradually decreased or are almost no longer used. Therefore, monitoring and optimizing the operation of SQL have become an important task in the operation and maintenance monitoring of big data platforms and are the basic work to ensure the healthy operation of the platform.

[0031] Figure 2 A flowchart of an embodiment of a big data platform operation and maintenance method of the present invention is shown as follows: Figure 2 As shown, the method includes the following steps:

[0032] S210: Collect Hive logs and YARN logs.

[0033] Because Hive logs are continuously appended files, Hive access in actual production systems typically requires multiple servers (HiveServers). A Hive log collection agent must be deployed on each HiveServer. Hive logs can be collected through a Flume cluster, a log collection system. Specifically, the Flume cluster agent configures the Hive log collection path to obtain Hive logs in real time and then collects them into a Kafka cluster.

[0034] Similarly, configure the Yarn log collection path through the Flume cluster's Agent to collect Yarn logs and then collect the Yarn logs into the Kafka cluster. In this step, the Hive log and Yarn log collection threads are executed independently. Therefore, Hive logs and Yarn logs can be collected in parallel through multiple independent threads, thereby improving the collection rate of Hive logs and Yarn logs and achieving efficient operation and maintenance of the big data platform.

[0035] S220: Parse the Hive log to obtain Hive execution information, and write the Hive execution information into the monitoring table.

[0036] Hive execution information includes: SQL instructions, query identifiers, SQL scripts, and execution start time.

[0037] Parse the Hive logs collected in step S210 to obtain Hive execution information, including the SQL instruction, query identifier (Query_ID), SQL script (SQL_Text), and execution start time. SQL_TEXT is the script that executes the SQL instruction, typically including SQL statements such as select and insert. The SQL instruction includes SQL_Text and system connection settings. In some cases, the SQL instruction and SQL_Text can be equivalent. Query_ID is the unique query identifier for each SQL instruction initiated by the Hive layer. The execution start time is the time the SQL instruction was submitted.

[0038] Furthermore, the above Hive execution information is written into a monitoring table. Specifically, the monitoring table may be a table (Hive_Monitor_Tab) of a MYSQL big data performance monitoring database. If the log has been executed, the status information is modified.

[0039] S230: Parse the Yarn log to obtain task execution information, and write the task execution information into the monitoring table.

[0040] In an optional manner, the task execution information includes: task execution account, task execution status, task read data volume, number of Map tasks, and number of Reduce tasks.

[0041] Specifically, based on the Query_ID obtained by parsing the Hive log, the task identifier (Job_ID) corresponding to each Query_ID is obtained, and then the MapReduce execution log is obtained from the Yarn log. The data corresponding to the data written in the monitoring table is found in the MapReduce execution log, thereby updating the monitoring table (that is, Hive_Monitor_Tab) with data such as the current task execution status, task read data volume, number of Map tasks, and number of Reduce tasks of each SQL being executed.

[0042] The task execution status includes submitted execution, executing, and / or completed; the task read data volume refers to the number of HDFS files read by Map; the number of Map tasks refers to the number of started Map tasks; and the number of Reduce tasks refers to the number of started Reduce tasks.

[0043] S240: Analyze and count Hive execution information to obtain SQL analysis and statistical results.

[0044] In an optional manner, step S240 further includes: analyzing the SQL script in the Hive execution information to obtain a target SQL script with the same SQL statement but different parameters; and performing statistics on the execution information of the target SQL script to obtain SQL analysis statistics results.

[0045] Because big data analysis development tools or SQL scripts (SQL_Text) typically repeatedly execute parts of SQL programs, analyzing SQL execution frequency is an important means of performance optimization. However, in big data development, some SQL development requires standardized conversion of SQL_Text due to different SQL_Text parameters and different SQL_Text collected in logs.

[0046] Therefore, in an optional embodiment of the present invention, analyzing the SQL scripts in the Hive execution information to obtain the target SQL script with the same SQL statement but different parameters further includes: converting the SQL script into a preset standard format, analyzing the SQL script after format conversion to obtain the SQL script to be processed with the same SQL statement but different parameters, and writing the SQL script to be processed into an SQL script statistics table; obtaining a pre-built standard SQL script data table containing multiple standard SQL scripts; for each SQL script to be processed in the SQL script statistics table, matching the SQL script to be processed with the standard SQL script in the standard SQL script data table, and determining the target SQL script based on the matching result.

[0047] Specifically, by classifying the SQL scripts executed every day, finding the parameter data, and establishing an SQL script statistics table, for the SQL scripts to be processed with the same SQL statements but different parameters in the Hive execution information, the SQL scripts to be processed are matched with the standard SQL scripts in the standard SQL script data table to determine the target SQL script.

[0048] Specifically, for two SQL scripts to be processed with the same SQL statement but different parameters, for example:

[0049] SQL script to be processed 1: select count(*) from tab_a where day_id = '2020-03-15';

[0050] SQL script 2 to be processed: select count(*) from tab_a where day_id = '2020-03-16'.

[0051] It can be seen that the subject content of the SQL statements in the two SQL scripts is the same, but the parameters (ie, day_id) brought in are different. The SQL script can be converted into a preset standard format. The converted standard SQL script can be: select count(*) from tab_a where day_id='$1'.

[0052] Taking the date parameter as an example, since the strings of commonly used SQL statements are relatively short, a polling method can be used to find substrings that conform to the date format from the string, identify the date field of the SQL script in the Hive execution information, replace the date field with a unified format, obtain the SQL script to be processed, and write it to the SQL script statistics table. Then, the SQL script to be processed can be matched with the standard SQL script.

[0053] Furthermore, a standard SQL script data table containing multiple standard SQL scripts is pre-constructed. In an optional manner, for each pending SQL script in the SQL script statistics table, the pending SQL script is matched with a standard SQL script in the standard SQL script data table, and determining the target SQL script based on the matching result further includes: determining whether the pending SQL script can be found in the standard SQL script data table; if so, determining the pending SQL script as the target SQL script; if not, calculating the Euclidean distance between the pending SQL script and each standard SQL script in the standard SQL script data table using a cosine similarity algorithm, and determining the pending SQL script whose Euclidean distance is less than a preset distance threshold as the target SQL script.

[0054] Specifically, for each SQL script to be processed, determine whether the SQL script to be processed can be queried in the standard SQL script data table; if so, determine the SQL script to be processed as the target SQL script; if not, calculate the Euclidean distance between the SQL script to be processed and each standard SQL script in the standard SQL script data table, and compare the similarity between the SQL script to be processed and each standard SQL script. If the distance is less than a preset distance threshold (for example, the preset distance threshold can be set to 1), it can be considered that the SQL script to be processed is very similar to the standard SQL script, then the SQL script to be processed can be determined as the target SQL script, and the standard SQL script data table can be updated according to the target SQL script as the basic data for statistics of commonly used target SQL scripts.

[0055] The process of calculating the Euclidean distance between the SQL script to be processed and each standard SQL script in the standard SQL script data table may include the following steps 1 to 3:

[0056] Step 1: Perform word segmentation on the SQL script to be processed. The complete sentence is divided into independent character sets according to the word segmentation algorithm. The frequency of each character in the character set is counted, and the word frequency vector x(x1, x2, ..., xn) of the SQL script to be processed is generated based on the frequency of each character. Here, x represents the word frequency vector of the SQL script to be processed, and x1, x2, ..., xn represent the frequencies of different characters in the character set.

[0057] Step 2: For each standard SQL script collected in the standard SQL script data table, perform word segmentation and count the frequency of each character to obtain the word frequency vector y(y1,y2,…,yn) of the standard SQL script, where y represents the word frequency vector of the standard SQL script and y1,y2,…,yn represent the frequencies of different characters in the character set.

[0058] Step 3: Calculate the Euclidean distance between the SQL script to be processed and the standard SQL script. The formula is as follows:

[0059]

[0060] S250: Output the monitoring table and SQL analysis and statistical results to the front-end page for display.

[0061] Specifically, administrators and other personnel can view the status and execution progress of all SQL statements currently executed by users by account, drill down on the currently executing SQL statement to query the task ID, task execution status, task read data volume, number of Map tasks, number of Reduce tasks, etc. Based on the SQL statement, they can view the execution time of completed tasks and other key execution data, thereby easily identifying SQL statement problems and promptly notifying optimizers to optimize them. In an optional embodiment, the method further includes: generating a SQL optimization plan based on the SQL analysis and statistical results, so that optimizers can refer to the SQL optimization plan to formulate a final SQL optimization strategy for optimizing the SQL statement.

[0062] Furthermore, step S240 is used to analyze the number of times the same SQL statement is executed, the number of resources used for each code execution, etc., thereby analyzing the execution frequency of the SQL statement. Based on the execution frequency of the SQL statement and the task execution information, SQL statements whose execution frequency exceeds a preset threshold or whose number of tasks started during execution is too large (for example, the number of started Map tasks and / or the number of started Reduce tasks exceeds the preset threshold) are monitored and optimized in a focused manner, thereby improving the overall performance of the system.

[0063] The method provided in this embodiment collects Hive logs and YARN logs; parses the Hive logs to obtain Hive execution information, and writes the Hive execution information into a monitoring table; parses the YARN logs to obtain task execution information, and writes the task execution information into a monitoring table; analyzes and counts the Hive execution information to obtain SQL analysis and statistical results; and outputs the monitoring table and SQL analysis and statistical results to a front-end page for display. This method collects and analyzes operation data through a big data platform, and then outputs the data to the front-end for display, assisting operation and maintenance personnel in performing SQL detection and optimization suggestions, thereby realizing efficient and intelligent operation and maintenance of the big data platform, thereby improving the operation efficiency of the program and the operation efficiency of the Hadoop big data platform, and solving the technical problem that existing operation and maintenance monitoring software and operation and maintenance monitoring methods cannot support daily optimization and that system performance may be slow due to some inefficient SQL.

[0064] Figure 3 The following is a schematic diagram showing the structure of an embodiment of a big data platform operation and maintenance device of the present invention. Figure 3 As shown, the device includes: a log collection module 310, a Hive log parsing module 320, a Yarn log parsing module 330, an analysis module 340, an output module 350 and an optimization module 360.

[0065] The log collection module 310 is used to collect Hive logs and Yarn logs.

[0066] The Hive log parsing module 320 is used to parse the Hive log, obtain Hive execution information, and write the Hive execution information into the monitoring table.

[0067] Hive execution information includes: SQL instructions, query identifiers, SQL scripts, and execution start time.

[0068] The Yarn log parsing module 330 is used to parse the Yarn log, obtain task execution information, and write the task execution information into the monitoring table.

[0069] In an optional manner, the task execution information includes: task execution account, task execution status, task read data volume, number of Map tasks, and number of Reduce tasks.

[0070] The analysis module 340 is used to analyze and count Hive execution information to obtain SQL analysis and statistical results.

[0071] In an optional manner, the analysis module 340 is further configured to analyze the SQL script in the Hive execution information to obtain a target SQL script with the same SQL statement but different parameters; and perform statistics on the execution information of the target SQL script to obtain SQL analysis statistics results.

[0072] In an optional manner, the analysis module 340 is further used to convert the SQL script into a preset standard format, analyze the SQL script after format conversion, obtain a pending SQL script with the same SQL statement but different parameters, and write the pending SQL script into an SQL script statistics table; obtain a pre-built standard SQL script data table containing multiple standard SQL scripts; for each pending SQL script in the SQL script statistics table, match the pending SQL script with the standard SQL script in the standard SQL script data table, and determine the target SQL script based on the matching result.

[0073] In an optional manner, the analysis module 340 is further used to determine whether the SQL script to be processed can be found in the standard SQL script data table; if so, the SQL script to be processed is determined as the target SQL script; if not, the cosine similarity algorithm is used to calculate the Euclidean distance between the SQL script to be processed and each standard SQL script in the standard SQL script data table, and the SQL script to be processed whose Euclidean distance is less than a preset distance threshold is determined as the target SQL script.

[0074] The output module 350 is used to output the monitoring table and SQL analysis and statistical results to the front-end page for display.

[0075] In an optional manner, the device further includes: an optimization module 360, configured to generate an SQL optimization solution based on SQL analysis statistics.

[0076] The device provided in this embodiment collects Hive logs and YARN logs; parses the Hive logs to obtain Hive execution information, and writes the Hive execution information into a monitoring table; parses the YARN logs to obtain task execution information, and writes the task execution information into a monitoring table; analyzes and counts the Hive execution information to obtain SQL analysis and statistical results; and outputs the monitoring table and SQL analysis and statistical results to a front-end page for display. The device collects and analyzes operation data through a big data platform, and outputs the data to the front-end for display, assisting operation and maintenance personnel in performing SQL detection and optimization suggestions, thereby realizing efficient and intelligent operation and maintenance of the big data platform, thereby improving the operation efficiency of the program and the operation efficiency of the Hadoop big data platform, and solving the technical problem that existing operation and maintenance monitoring software and operation and maintenance monitoring methods cannot support daily optimization and that system performance may be slow due to some inefficient SQL.

[0077] An embodiment of the present invention provides a non-volatile computer storage medium, which stores at least one executable instruction. The computer executable instruction can execute the big data platform operation and maintenance method in any of the above method embodiments.

[0078] The executable instructions can be used to cause the processor to perform the following operations:

[0079] Collect Hive logs and YARN logs;

[0080] Parse Hive logs to obtain Hive execution information and write the Hive execution information into the monitoring table;

[0081] Parse the Yarn log to obtain task execution information and write the task execution information into the monitoring table;

[0082] Analyze and count Hive execution information to obtain SQL analysis and statistical results;

[0083] Output the monitoring table and SQL analysis and statistical results to the front-end page for display.

[0084] Figure 4 The schematic diagram of the structure of the computing device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0085] like Figure 4 As shown, the computing device may include:

[0086] Processor, Communications Interface, Memory, and Communication Bus.

[0087] The processor, communication interface, and memory communicate with each other via a communication bus. The communication interface is used to communicate with other devices, such as client devices or other server network elements. The processor is used to execute programs, specifically, the steps described in the aforementioned big data platform operation and maintenance method embodiment.

[0088] Specifically, the program may include program codes including computer operation instructions.

[0089] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the server may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0090] Memory is used to store programs. The memory may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk storage.

[0091] The program can be specifically used to cause the processor to perform the following operations:

[0092] Collect Hive logs and YARN logs;

[0093] Parse Hive logs to obtain Hive execution information and write the Hive execution information into the monitoring table;

[0094] Parse the Yarn log to obtain task execution information and write the task execution information into the monitoring table;

[0095] Analyze and count Hive execution information to obtain SQL analysis and statistical results;

[0096] Output the monitoring table and SQL analysis and statistical results to the front-end page for display.

[0097] The algorithm or demonstration provided herein are not inherently relevant to any particular computer, virtual system or other equipment. Various general-purpose systems may also be used together with the teachings based on this. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the above description of specific languages is for the purpose of disclosing the best mode of the present invention.

[0098] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0099] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0100] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0101] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0102] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to an embodiment of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0103] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A big data platform operation and maintenance method, characterized in that: include: Collect Hive logs and YARN logs; Parse Hive logs to obtain Hive execution information, and write the Hive execution information into the monitoring table; The Hive execution information includes: SQL instructions, query identifiers, SQL scripts, and execution start time; Parse the Yarn log to obtain task execution information, and write the task execution information into the monitoring table; the task execution information includes: task execution account, task execution status, task read data volume, number of Map tasks and number of Reduce tasks; Analyze and count the Hive execution information to obtain SQL analysis and statistical results; Output the monitoring table and the SQL analysis statistical results to the front-end page for display; The step of analyzing and counting the Hive execution information to obtain SQL analysis and statistics results further includes: Analyze the SQL script in the Hive execution information to obtain a target SQL script with the same SQL statement but different parameters; wherein the target SQL script is determined by analyzing the SQL script converted into a preset standard format to obtain a pending SQL script with the same SQL statement but different parameters, and matching the pending SQL script with a standard SQL script in a standard SQL script data table; The execution information of the target SQL script is counted to obtain SQL analysis statistical results.

2. The method according to claim 1, characterized in that The analyzing the SQL script in the Hive execution information to obtain a target SQL script with the same SQL statement but different parameters further includes: Writing the SQL script to be processed into the SQL script statistics table; Get a pre-built standard SQL script data table containing multiple standard SQL scripts; For each SQL script to be processed in the SQL script statistics table, the SQL script to be processed is matched with a standard SQL script in the standard SQL script data table, and a target SQL script is determined according to the matching result.

3. The method according to claim 2, characterized in that The step of matching each SQL script to be processed in the SQL script statistics table with a standard SQL script in the standard SQL script data table, and determining a target SQL script according to the matching result further comprises: Determine whether the SQL script to be processed can be found in the standard SQL script data table; If yes, the SQL script to be processed is determined as the target SQL script; If not, the cosine similarity algorithm is used to calculate the Euclidean distance between the SQL script to be processed and each standard SQL script in the standard SQL script data table, and the SQL script to be processed whose Euclidean distance is less than a preset distance threshold is determined as the target SQL script.

4. The method according to any one of claims 1 to 3, characterized in that After analyzing and counting the Hive execution information to obtain SQL analysis and statistical results, the method further includes: Generate an SQL optimization plan based on the SQL analysis and statistical results.

5. A big data platform operation and maintenance device, characterized in that: include: Log collection module, used to collect Hive logs and Yarn logs; A Hive log parsing module is used to parse Hive logs, obtain Hive execution information, and write the Hive execution information into a monitoring table; the Hive execution information includes: SQL instructions, query identifiers, SQL scripts, and execution start time; A Yarn log parsing module is used to parse Yarn logs, obtain task execution information, and write the task execution information into the monitoring table; the task execution information includes: task execution account, task execution status, task read data volume, number of Map tasks, and number of Reduce tasks; An analysis module is used to analyze and count the Hive execution information to obtain SQL analysis and statistical results; Output module, used to output the monitoring table and the SQL analysis and statistical results to the front-end page for display; Wherein, the analysis module is further used for: Analyze the SQL script in the Hive execution information to obtain a target SQL script with the same SQL statement but different parameters; wherein the target SQL script is determined by analyzing the SQL script converted into a preset standard format to obtain a pending SQL script with the same SQL statement but different parameters, and matching the pending SQL script with a standard SQL script in a standard SQL script data table; The execution information of the target SQL script is counted to obtain SQL analysis statistical results.

6. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the big data platform operation and maintenance method according to any one of claims 1 to 4.

7. A computer storage medium, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to perform an operation corresponding to a big data platform operation and maintenance method as described in any one of claims 1-4.

Images