A large task killing method, system, device and medium based on an OLAP component

Abstract

The application discloses a large task killing method, system, device and medium based on an OLAP component, obtains query information of a cluster through the OLAP component, analyzes and judges a large task according to the query information, and thus executes an execution task type of the OLAP component corresponding to the cluster containing the large task in the query information, guarantees the query performance, availability and stability of the cluster and the business, can automatically kill or alarm the large task in the cluster in the case of multiple clusters and multiple components, and a unified configuration mode is more convenient for users to manage the OLAP component and the cluster, reduces operation and maintenance complexity and cost, and solves the technical problems that a traditional large task killing system is often designed for specific clusters or components, lacks unified configuration and management modes, increases operation and maintenance complexity, improves error risk, and is low in human configuration efficiency and high in cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a large task killing method and system based on an OLAP component, a device, and a medium. BACKGROUND

[0002] OLAP (On-line Analytical Processing) is a key data analysis technology that allows users to quickly analyze and summarize data from multiple perspectives. This technology is particularly suitable for processing and analyzing large amounts of data, such as information in enterprise data warehouses. OLAP systems usually support complex queries, including operations such as slicing, dicing, summarizing, and comparing multi-dimensional data.

[0003] With the advent of the big data era, the amount of data that organizations need to process has increased dramatically, which requires more powerful OLAP systems to meet analysis needs. Modern OLAP components such as ClickHouse, Presto, and Doris have emerged to provide high-performance data processing capabilities, enabling real-time analysis and fast queries to meet the needs of different business scenarios.

[0004] However, as OLAP cluster scales and business diversifies, a single large task can have a significant impact on the performance of the entire cluster, leading to resource shortages and even unstable services. In this case, a large task killing system is particularly important. This system can monitor the load of the cluster and automatically take measures such as limiting resource usage, delaying execution, or terminating tasks when a large task is detected to protect the stability and availability of the cluster.

[0005] Traditional large task killing systems are often designed for specific clusters or components, lacking a unified configuration and management approach. This not only increases the complexity of operation and maintenance, but also increases the risk of errors. In a multi-component, multi-cluster environment, operation and maintenance personnel need to manage a large number of configurations, which is not only inefficient but also costly. SUMMARY

[0006] The present application provides a large task killing method and system based on an OLAP component, a device, and a medium, solving the technical problem of traditional large task killing systems that are often designed for specific clusters or components, lack a unified configuration and management approach, increase the complexity of operation and maintenance, and increase the risk of errors, and the inefficiency and high cost of manual configuration.

[0007] Therefore, the first aspect of the present application provides a large task killing method based on an OLAP component, the method comprising:

[0008] obtaining query information of a cluster based on an OLAP component;

[0009] determining whether the query information is a large task based on the query information of the cluster, if yes, retaining the query information, otherwise filtering the query information;

[0010] determining a task type based on the retained query information, and executing a task instruction according to the task type, wherein the task type includes a killing task and an alarm task;

[0011] if the task type is the killing task, the task instruction is to connect the cluster corresponding to the query information according to interface information contained in the query information, and to call a preset killing instruction for killing;

[0012] if the task type is the alarm task, the task instruction is to save the query information to a metric interface, and to collect the query information by a monitoring alarm system and perform alarm processing.

[0013] Optionally, the obtaining of the query information of the cluster based on the OLAP component specifically includes:

[0014] obtaining collection configuration information of different OLAP components;

[0015] generating a cluster query instruction of each of the OLAP components according to the collection configuration information;

[0016] sending the generated cluster query instruction to each of the OLAP components, executing cluster query by each of the OLAP components, and returning the query information of the cluster.

[0017] Optionally, the obtaining of the query information of the cluster based on the OLAP component further includes:

[0018] combining each of the OLAP components with a large-scale parallel processing computing framework;

[0019] generating a query plan according to a data distribution condition of a cluster corresponding to the OLAP component based on a query optimizer of the large-scale parallel processing computing framework.

[0020] Optionally, the combination of each of the OLAP components with the large-scale parallel processing computing framework further includes:

[0021] based on the large-scale parallel processing computing framework, the OLAP component decomposes a received cluster query instruction into a plurality of subtasks;

[0022] based on the large-scale parallel processing computing framework, the plurality of subtasks are executed in parallel on a plurality of nodes.

[0023] Optionally, the determining whether the query information is a large task based on the query information of the cluster comprises:

[0024] parsing the query information of the cluster according to the component type of the OLAP component;

[0025] determining whether the query information is a large task according to the parsed query information and the component type of the OLAP component, and if yes, retaining the query information, otherwise filtering the query information.

[0026] Optionally, if the task type is a killing task, the task instruction comprises connecting the cluster corresponding to the query information according to the interface information contained in the query information, and calling a preset killing instruction to kill.

[0027] saving the query information to a metric interface;

[0028] a monitoring and alarming system collecting the query information from the metric interface and performing alarm processing.

[0029] Optionally, the monitoring and alarming system collecting the query information from the metric interface and performing alarm processing comprises:

[0030] defining a preset alarm rule based on the key performance indicators of the query information and a preset threshold;

[0031] collecting the query information from the metric interface in real time;

[0032] triggering an alarm when it is detected that the key performance indicators of the query information exceed the preset threshold;

[0033] performing alarm response processing.

[0034] The second aspect of the present application provides a large task killing system based on an OLAP component, the system comprising:

[0035] a collection module, an analysis module and a configuration module;

[0036] the collection module is configured to obtain query information of a cluster based on an OLAP component, and send the query information to the analysis module;

[0037] the analysis module is configured to determine whether the query information is a large task based on the query information of the cluster, and if yes, retain the query information, otherwise filter the query information, and send the retained query information to the configuration module;

[0038] The configuration module is configured to determine a task type based on the query information, if the task type is a killing task, connect the cluster corresponding to the query information according to interface information contained in the query information, and call a preset killing instruction to kill, if the task type is an alarm task, save the query information to a metric interface, collect the query information by a monitoring alarm module and perform alarm processing.

[0039] Optionally, the collection module is specifically configured to:

[0040] Obtain collection configuration information of different OLAP components;

[0041] Generate a cluster query instruction of each of the OLAP components according to the collection configuration information;

[0042] Send the generated cluster query instruction to each of the OLAP components, execute cluster query by each of the OLAP components, and return query information of the cluster.

[0043] Optionally, the collection module is further configured to:

[0044] Combine each of the OLAP components with a large-scale parallel processing computing framework;

[0045] Generate a query plan according to data distribution of a cluster corresponding to the OLAP component based on a query optimizer of the large-scale parallel processing computing framework.

[0046] Optionally, the collection module is further configured to:

[0047] Based on the large-scale parallel processing computing framework, the OLAP component decomposes a received cluster query instruction into a plurality of subtasks;

[0048] Based on the large-scale parallel processing computing framework, the plurality of subtasks are executed in parallel on a plurality of nodes.

[0049] Optionally, the analysis module is specifically configured to:

[0050] According to a component type of each of the OLAP components, analyze the query information of the cluster;

[0051] According to the analyzed query information, in combination with the component type of the OLAP component, call a judgment instruction corresponding to the component type to determine whether the query information is a large task, if yes, retain the query information, otherwise, filter the query information.

[0052] Optionally, further comprising:

[0053] The monitoring alarm module is configured to collect the query information stored in the metric interface and perform alarm processing.

[0054] Optionally, the monitoring alarm is specifically configured to:

[0055] defining a preset alarm rule based on a key performance indicator of the query information and a preset threshold value;

[0056] collecting the query information from the metric interface in real time;

[0057] triggering an alarm when it is detected that the key performance indicator of the query information exceeds the preset threshold value;

[0058] performing alarm response processing.

[0059] The third aspect of the present application provides a large task killing device based on an OLAP component, the device comprising a processor and a memory:

[0060] The memory is configured to store program code and transmit the program code to the processor;

[0061] The processor is configured to execute the steps of the method for killing large tasks based on the OLAP component according to the instructions in the program code.

[0062] The fourth aspect of the present application provides a computer readable storage medium for storing program code, the program code being configured to execute the method of the first aspect.

[0063] From the above technical solutions, the embodiments of the present application have the following advantages:

[0064] In the present application, a large task killing method, system, device and medium based on an OLAP component are provided, query information of a cluster is obtained through an OLAP component, and a large task is analyzed and determined according to the query information, so that the execution task type of the OLAP component corresponding to the cluster containing the large task in the query information is executed, the query performance, availability and stability of the cluster and the business are guaranteed; in the case of multiple clusters and multiple components, the large task in the cluster can be automatically killed or alarmed, and the unified configuration mode is more convenient for users to manage the OLAP component and the cluster, reduces the operation and maintenance complexity and cost, and solves the technical problems of the traditional large task killing system, which is often designed for specific clusters or components, lacks unified configuration and management mode, increases the complexity of operation and maintenance, improves the risk of error, and is low in artificial configuration efficiency and high in cost. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1A method flowchart of the large task killing method based on the OLAP component in the embodiment of the present application is provided.

[0066] Figure 2 A system architecture diagram of the large task killing system based on the OLAP component in the embodiment of the present application is provided.

[0067] Figure 3 A structural schematic diagram of the large task killing device based on the OLAP component in the embodiment of the present application is provided. DETAILED DESCRIPTION

[0068] In order to enable personnel in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0069] The present application designs a large task killing method, system, device and medium based on an OLAP component, solves the technical problems of the traditional large task killing system, which is often designed for specific clusters or components, lacks unified configuration and management mode, increases the complexity of operation and maintenance, improves the risk of error, and low efficiency and high cost of manual configuration.

[0070] For the convenience of understanding, please refer to Figure 1 , Figure 1 A method flowchart of the large task killing method based on the OLAP component in the embodiment of the present application is provided, as shown in Figure 1 , specifically comprising:

[0071] S1, obtaining query information of a cluster based on an OLAP component;

[0072] Specifically comprising:

[0073] Obtaining collection configuration information of different OLAP components;

[0074] Generating cluster query instructions of each OLAP component according to the collection configuration information;

[0075] Sending the generated cluster query instructions to each OLAP component, executing cluster query by each OLAP component, and returning the query information of the cluster.

[0076] It should be noted that the list of OLAP components that need to be monitored is determined, such as ClickHouse, Presto, Doris, Druid, etc.

[0077] Configure data collection tools or agents for each component to collect query information of the cluster.

[0078] Ensure that the collection tools can obtain key information including query statements, execution time, resource usage, etc.

[0079] The overall logic of different components is consistent, and the implementation is adjusted according to different components. For example, the logic of obtaining key information of the cluster, presto is to call the interface to obtain the query Query information, and doris is to call the interface to obtain.

[0080] This application takes presto as an example to illustrate an application example. According to the collection configuration information, for example, by configuring action.presto_query_kill.analyze_period, the unit is millisecond, configure the time interval of calling the interface of presto.

[0081] For example, 10 seconds, call the interface of presto every 10 seconds to obtain the query Query information of the cluster. Query information package includes SQL statement, SQL execution time, current memory usage, cumulative memory usage, query ID and other information.

[0082] Each different OLAP component can obtain different query Query information, as shown in the following table:

[0083]

[0084] Further, based on the OLAP component to obtain the query information of the cluster also includes:

[0085] Combine each OLAP component with a large-scale parallel processing computing framework;

[0086] Based on the query optimizer of the large-scale parallel processing computing framework, generate a query plan according to the data distribution of the OLAP component corresponding to the cluster.

[0087] Further, after combining each OLAP component with a large-scale parallel processing computing framework, it also includes:

[0088] Based on the large-scale parallel processing computing framework, the OLAP component decomposes the received cluster query instruction into several subtasks;

[0089] Based on the large-scale parallel processing computing framework, execute the subtasks in parallel on several nodes.

[0090] It is worth noting that combining OLAP components with MPP (Massively Parallel Processing) computing frameworks can improve the query performance of OLAP systems, especially when dealing with large-scale datasets. Specifically, OLAP components can be integrated into MPP computing frameworks, enabling query operations to be executed in parallel across multiple computing nodes.

[0091] The query optimizer analyzes the query request and generates an efficient query execution plan based on the distribution of data across the cluster. This plan determines how to break down the query, how data is transferred, and how computing resources are allocated.

[0092] The OLAP component breaks down the query into multiple subtasks, each of which can be executed independently, based on data partitioning, sharding, or other logical criteria.

[0093] In the MPP computing framework, subtasks are distributed to different nodes in the cluster. Each node independently executes the subtasks assigned to it, and can do so simultaneously, significantly reducing the overall query processing time.

[0094] Suppose we have a large data warehouse stored in an MPP computing framework consisting of 100 nodes. We want to execute a complex analytical query, such as calculating the annual sales total for each region.

[0095] Query reception: The user submits a query request through an OLAP frontend tool.

[0096] Query optimization: The query optimizer in the MPP computing framework analyzes the query and generates a query plan considering the distribution of data across the nodes.

[0097] Task decomposition: The query is broken down into multiple subtasks, such as calculating sales data for each region on different nodes.

[0098] Parallel execution: Each node processes its assigned subtasks in parallel, calculating the annual sales total for its respective region.

[0099] Result aggregation: After each node completes its calculations, it sends the results back to the coordinator node, which is responsible for combining the results of all subtasks to form the final query result.

[0100] Result return: The coordinator node returns the aggregated results to the user.

[0101] Furthermore, automated configuration management tools can be used to simplify the collection and configuration of different components. Standardized query information formats can also be used to facilitate subsequent processing and analysis.

[0102] Further, the plugin or adapter pattern can be utilized to easily extend support for new OLAP components without modifying the existing code base.

[0103] The plugin or adapter pattern defines:

[0104] Plugin pattern: Allows the system to dynamically load and unload modules at runtime, which implement specific interfaces to provide additional functionality.

[0105] Adapter pattern: Ensures that different OLAP components can be integrated with the monitoring system in a uniform manner, even if their data access interfaces and data formats differ.

[0106] Design a uniform interface:

[0107] Define a common data collection interface that all plugins or adapters must implement. This way, regardless of the specific implementation of the OLAP component, the monitoring system can interact with it in a uniform manner.

[0108] Develop specific adapters:

[0109] Develop specific adapters for each OLAP component, which implement the uniform interface but have internal logic optimized for the specific component to ensure efficient data collection.

[0110] No need to modify existing code:

[0111] Through the plugin or adapter pattern, adding support for new OLAP components does not require modifying the existing code of the monitoring system. This reduces maintenance costs and improves development efficiency.

[0112] Easy to extend and maintain:

[0113] When support for new OLAP components is needed, only a new adapter or plugin needs to be developed and integrated into the system without the need for extensive modifications to the existing code.

[0114] Further, introduce a load balancing mechanism for data collection to ensure that data collection does not become a bottleneck for OLAP components under high concurrency.

[0115] Under high concurrency, ensure that data collection operations do not negatively impact the performance of OLAP components and avoid becoming a system bottleneck.

[0116] Implementation strategy:

[0117] Distributed collection: Distribute data collection tasks to multiple collection nodes, with each node responsible for a portion of the data collection work, thereby dispersing the load.

[0118] Sampling and Aggregation: During data collection, sampling and aggregation techniques are used to reduce the amount of data that needs to be transmitted and processed.

[0119] Priority Scheduling: Different priorities are set for data collection tasks based on the importance and real-time requirements of the data, ensuring that critical data is collected first.

[0120] Monitoring and Adjustment:

[0121] Real-time monitoring of data collection load and performance, dynamically adjusting collection strategies and resource allocation according to actual conditions.

[0122] Failover and Redundancy:

[0123] Implementing failover and redundancy mechanisms between collection nodes to ensure high availability and reliability of data collection.

[0124] Resource Optimization:

[0125] Optimizing resource allocation for data collection, such as CPU, memory, and network bandwidth, based on resource usage and performance indicators of OLAP components.

[0126] S2, based on the cluster query information, determine whether the query information is a large task, if so, keep the query information, otherwise filter the query information;

[0127] Specifically including:

[0128] According to the component type of different OLAP components, the query information of the cluster is parsed;

[0129] According to the parsed query information, combined with the component type of the OLAP component, the judgment instruction corresponding to the component type is called to determine whether the query information is a large task, if so, keep the query information, otherwise filter the query information.

[0130] It should be noted that the collected query information is parsed, and key indicators such as execution time and memory usage are extracted. According to the preset threshold or rule, it is determined whether the query meets the conditions of a large task. For large tasks that meet the conditions, their query information is retained; otherwise, it is filtered.

[0131] This application takes Presto as an example to illustrate an application example. Taking 10 seconds as an example, the collected Presto query information is obtained every 10 seconds, and the query information includes the value of SQL execution time, the value of cumulative memory usage, and the value of current memory usage;

[0132] Based on the query information, it is determined whether it is a large task. When the value of SQL execution time consumption, the value of cumulative memory usage, and the value of current memory usage all exceed the corresponding threshold, the query information is a large task, and the query information is retained. Otherwise, the query information is filtered.

[0133] Further, based on intelligent analysis tools, historical data and machine learning algorithms can be used to optimize the determination criteria of large tasks, realize dynamic threshold adjustment, and automatically adjust the determination criteria according to system load and performance.

[0134] S3, determine the task type based on the retained query information, and execute the task instruction according to the task type, wherein the task type includes a killing task and an alarm task;

[0135] It should be noted that the retained query information is classified to determine its task type, such as a killing task or an alarm task. According to the task type, corresponding processing strategies and instructions are formulated.

[0136] In the monitoring system of the OLAP component, determining the task type based on the retained query information and executing the corresponding task instruction is a key process. This process involves analysis of query information, determination of task type, and execution of corresponding operations. The following is a specific explanation:

[0137] Analysis of query information:

[0138] The monitoring system first collects and analyzes query information from the OLAP component. These information may include query execution time, resource consumption (such as CPU, memory usage), query source, query complexity, etc.

[0139] Determine the task type:

[0140] According to the analysis results, the system needs to determine whether each query information triggers the preset task type condition. The task type usually includes:

[0141] Killing task: when the query consumes resources exceeding the preset threshold, or the query execution time is too long, which may have a negative impact on system performance, the system will mark it as a killing task.

[0142] Alarm task: when the query shows abnormal behavior, but has not reached the level that needs to be immediately killed, the system may mark it as an alarm task to remind the administrator to further check.

[0143] Further, a more flexible task type definition mechanism can be designed to adapt to different business needs and scenarios, to realize automatic identification and classification of task types, and to reduce manual intervention. Specifically:

[0144] Design a more flexible task type definition mechanism, aiming to enable the monitoring system to automatically adapt to different business needs and scenarios, reducing manual intervention by automatically identifying and classifying task types. Here are the specific steps and methods to achieve this goal:

[0145] Define task characteristics:

[0146] Determine the key characteristics and indicators of different task types, such as query execution time, resource consumption, query frequency, etc. These characteristics will be used to distinguish between kill tasks and alert tasks.

[0147] Use machine learning algorithms:

[0148] Use machine learning algorithms such as classification algorithms (decision trees, random forests, support vector machines, etc.) to train models based on historical data to automatically identify and classify task types.

[0149] Through continuous learning and optimization, the model can adapt to new business scenarios and changes, improving the accuracy of task classification.

[0150] Real-time monitoring and analysis:

[0151] Implement a real-time monitoring system to continuously collect query information and performance indicators for timely analysis and processing.

[0152] Use stream processing technologies (such as Apache Kafka, Apache Flink, etc.) to process real-time data streams, improving data processing efficiency and response speed.

[0153] Dynamic threshold adjustment:

[0154] Set dynamic thresholds that automatically adjust based on system load, historical data, and real-time performance indicators. This helps maintain reasonable monitoring sensitivity in different business scenarios.

[0155] Dynamic thresholds can be achieved through machine learning model prediction or rule-based adaptive algorithms.

[0156] Task type automatic identification:

[0157] Based on predefined characteristics and dynamic thresholds, the monitoring system automatically determines whether query information meets the conditions of kill tasks or alert tasks.

[0158] The system can set multiple levels of thresholds to distinguish different severity tasks, such as mild alerts, moderate alerts, and severe kills.

[0159] Task classification and execution:

[0160] Once the task type is automatically identified, the system will execute the corresponding task instruction according to the task type. For example, for the kill task, the system will terminate or limit the resource consumption of the query; for the alarm task, the system will record the event and notify the administrator.

[0161] The system can set multiple levels of task instructions to adapt to tasks of different severity.

[0162] This application takes Presto as an example to illustrate an application example. For example, every 10 seconds, the retained query information is obtained, and the task type is determined according to the action.presto_query_kill.type configured in the query information. When action.presto_query_kill.type is find_and_kill, it is a kill task, and when action.presto_query_kill.type is find_and_warn, it is an alarm task.

[0163] S4, if the task type is a kill task, the task instruction is to connect the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset kill instruction for killing;

[0164] It should be noted that when the task type is determined to be a "kill task", it means that the monitoring system has identified that some query operations may have a negative impact on the performance of the OLAP component, such as long-term occupation of a large amount of resources or causing system instability. In this case, the system needs to take measures to terminate these queries to restore system performance and stability. The following are the specific execution steps of the "kill task":

[0165] Get interface information:

[0166] Extract interface information from query information collected by the monitoring system. These information usually includes the connection details of the database, such as host address, port number, authentication information, etc., which are necessary to connect to the OLAP component cluster.

[0167] Establish connection:

[0168] Using the extracted interface information, the monitoring system establishes a connection with the OLAP component cluster through the corresponding client or API. This step ensures that the monitoring system can directly interact with the target cluster.

[0169] Call kill instruction:

[0170] Once the connection is established, the monitoring system will call the preset kill instruction. The kill instruction is a command specially designed to terminate or limit those query operations that are identified as resource consumption.

[0171] For example, in Presto, a KILL QUERY command can be used to terminate an executing query; in ClickHouse, KILL QUERY WHERE query_id = 'xxxx' can be used to terminate a query with a specific ID.

[0172] Executing the kill:

[0173] The system executes the kill instruction, which will cause the target query to be terminated. This process may involve sending an interrupt signal to the query process or manipulating the cluster management interface.

[0174] Confirming the kill result:

[0175] After the kill operation is completed, the monitoring system needs to confirm whether the kill was successfully executed. This can be achieved by checking the query status, resource release, or query execution logs.

[0176] Recording and notification:

[0177] Regardless of whether the kill operation was successful, the monitoring system should record this event, including the reason for the kill, the time, the operation performed, and the result. In addition, the system should notify the relevant operations team or system administrator so that they can take further action, such as analyzing the reason for the kill and optimizing the query.

[0178] Resource recycling:

[0179] After the kill operation, the system should ensure that the previously occupied resources (such as memory, CPU time, etc.) are properly recycled so that other queries can utilize these resources.

[0180] Audit and compliance:

[0181] In some cases, the kill operation may need to comply with specific audit and compliance requirements. The monitoring system should ensure that the execution of the kill operation complies with these requirements and keep the corresponding records for auditing.

[0182] In this way, the kill task helps maintain the performance and stability of the OLAP component, preventing a single query from excessively consuming resources and affecting the overall operation of the system.

[0183] S5, if the task type is a kill task, the task instruction is to connect the cluster corresponding to the query information according to the interface information contained in the query information, and then call a preset kill instruction to kill, and further include:

[0184] Saving query information to a metric interface;

[0185] The monitoring and alarm system collects query information from the metric interface and performs alarm processing.

[0186] It should be noted that for the killing task, the interface information in the query information is connected to the corresponding cluster. The preset killing instruction is called to terminate or limit the resources of the large task.

[0187] Further, the monitoring and alarm system collects query information from the metric interface and performs alarm processing, specifically including:

[0188] Defining a set of alarm rules based on the key performance indicators (KPIs) of the query information and preset thresholds;

[0189] Real-time collection of query information from the metric interface;

[0190] When the key performance indicators of the query information are detected to exceed the preset thresholds, an alarm is triggered;

[0191] Performing alarm response processing.

[0192] It should be noted that a set of alarm rules are defined, which are based on the key performance indicators (KPIs) of the query information and preset thresholds;

[0193] Real-time monitoring of query information, when any indicator is detected to exceed the preset threshold, an alarm is triggered;

[0194] After the alarm is triggered, the system automatically executes a series of predefined response measures, such as sending notifications to system administrators, recording event logs, starting automatic diagnostic programs, or performing automatic repair operations;

[0195] During the alarm processing process, the system can take different levels of response measures according to the severity and urgency levels of the alarm;

[0196] The system supports customizing alarm notification methods, including but not limited to email, SMS, application push notifications, etc., to ensure timely notification to relevant personnel;

[0197] Providing a user interface for system administrators to view alarm history, current alarm status, and configure and manage alarm rules.

[0198] Customizing monitoring strategies according to the characteristics of OLAP components is a complex but crucial process that can help ensure system performance and stability. Here are some key steps and considerations for customizing effective monitoring strategies for different OLAP components:

[0199] Understand component characteristics:

[0200] First, it is necessary to have a deep understanding of the architecture, performance characteristics, common query patterns, and resource consumption characteristics of each OLAP component. For example, some components may be better at handling real-time queries, while others may be more suitable for batch processing.

[0201] Define Key Performance Indicators (KPIs):

[0202] Determine which metrics are critical for monitoring. Common KPIs include query response time, resource usage (CPU, memory, disk I / O), concurrent query count, query throughput, etc.

[0203] Configure Data Collection:

[0204] Configure data collection tools according to the characteristics of the component, ensuring that all critical performance data can be collected. This may include configuring API calls, log file parsing, or using specific monitoring tools.

[0205] Set Thresholds and Alerts:

[0206] Set thresholds for each critical metric, triggering alerts when the metric exceeds these thresholds. Thresholds should be set based on the component's performance baseline and business requirements.

[0207] Consider Query Patterns:

[0208] Analyze the query patterns of the component, identifying which queries are common and which are unusual. This helps set more accurate monitoring rules, such as different monitoring strategies for specific types of queries.

[0209] Implement Dynamic Adjustment:

[0210] Use machine learning or other automated tools to dynamically adjust monitoring strategies. For example, automatically adjust thresholds based on historical data and real-time performance metrics.

[0211] Integrate Massively Parallel Processing (MPP) Characteristics:

[0212] If the OLAP component is an MPP architecture, the monitoring strategy should reflect the distributed nature of the cluster. For example, monitor data distribution, load balancing between nodes, and cross-node query execution.

[0213] Consider Data Security and Privacy:

[0214] Ensure compliance with data security and privacy regulations during monitoring. For sensitive data, encryption or anonymization may be required.

[0215] User Customization and Extensibility:

[0216] Provide the ability for users to customize monitoring rules, allowing them to adjust monitoring strategies according to their specific needs.

[0217] Testing and Validation:

[0218] Before implementing the monitoring strategy, conduct thorough testing and validation to ensure that the monitoring tools and strategy can correctly capture and report performance issues.

[0219] Documentation and Training:

[0220] Provide detailed documentation for the monitoring strategy and provide necessary training for the operations team to ensure they understand the monitoring tools and strategy.

[0221] Continuous Optimization:

[0222] The monitoring strategy is not static and needs to be continuously optimized and adjusted according to changes in the system and business needs.

[0223] By following the above steps, we can customize effective monitoring strategies for different OLAP components, help to timely discover and solve performance problems, and improve the reliability and efficiency of the system.

[0224] Further, we can dynamically adjust the killing strategy according to the task characteristics and cluster state.

[0225] Further, we can design priority and queue mechanisms for different types of tasks to ensure that critical tasks can be processed first.

[0226] Further, we can provide rollback mechanisms for killing tasks to quickly recover when killing operations cause problems.

[0227] If the task type is an alarm task, the task instruction is to save the query information to the metric interface, and the monitoring alarm system collects the query information and performs alarm processing.

[0228] It should be noted that for alarm tasks, the query information is saved to the metric interface. The monitoring alarm system collects and processes the information for alarm processing.

[0229] Integrate multiple alarm channels such as email, SMS, real-time messages, etc. to ensure timely notification of relevant personnel.

[0230] Implement automated analysis and response suggestions for alarm information to help quickly locate problems and take action.

[0231] In addition, custom alarm rules and triggers can be developed to meet different monitoring needs. At the same time, we can provide visual display of alarm information such as dashboards and alarm boards, realize visual display and historical analysis of alarm information, and help the operations team better understand the system status and trends.

[0232] This application takes Presto as an example to illustrate an application example. For example, every 10 seconds, we get the large task query information of Presto.

[0233] When action.presto_query_kill.type configured in the query information is equal to find_and_kill, Presto cluster is connected according to presto_jdbc information configured in the query information, and allsystem.runtime.kill_query() command is called to kill the corresponding large task query information.

[0234] When action.presto_query_kill.type configured in the query information is equal to find_and_warn, no killing processing is performed.

[0235] Irrespective of whether the query information performs killing processing or not, the query information is output to a metric interface, and the query information is collected by a monitoring and alarming system for a preset time length, for example, 120 seconds, and the monitoring and alarming system performs mail alarming, telephone alarming or short message alarming or other alarming behaviors on the corresponding user according to the query information and the corresponding cluster.

[0236] Please refer to Figure 2 , Figure 2 The system architecture diagram of the large task killing system based on the OLAP component in the embodiment of the application is shown in Figure 2 , and specifically comprises:

[0237] a collection module 201, an analysis module 202 and a configuration module 203;

[0238] The collection module 201 is used for acquiring query information of a cluster based on an OLAP component, and sending the query information to the analysis module 202;

[0239] The analysis module 202 is used for judging whether the query information is a large task based on the query information of the cluster, if yes, retaining the query information, otherwise, filtering the query information, and sending the retained query information to the configuration module 203;

[0240] The configuration module 203 is used for determining a task type based on the query information, if the task type is a killing task, connecting the cluster corresponding to the query information according to interface information contained in the query information, and calling a preset killing instruction to kill, if the task type is an alarming task, saving the query information to a metric interface, collecting the query information by a monitoring and alarming module and performing alarming processing.

[0241] Another embodiment of the application provides a large task killing device based on an OLAP component, as shown in Figure 3 The device 10 comprises:

[0242] one or more processors 110 and a memory 120, Figure 3The processor 110 and the memory 120 can be connected through a bus or other means, Figure 3 The processor 110 and the memory 120 can be connected through a bus or other means,

[0243] The processor 110 is configured to complete various control logics of the device 10, and can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination of these components. In addition, the processor 110 can also be any conventional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP and / or any other such configuration.

[0244] The memory 120 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the construction method of the multi-lingual phoneme representation model in the embodiment of the application. The processor 110 executes the non-volatile software programs, instructions and units stored in the memory 120, thereby performing various function applications and data processing of the device 10, that is, implementing the construction method of the multi-lingual phoneme representation model in the above method embodiment.

[0245] The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the device 10, etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 can optionally include a memory remotely arranged with respect to the processor 110, and these remote memories can be connected to the device 10 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0246] One or more units are stored in the memory 120, and when executed by the one or more processors 110, the following steps are implemented:

[0247] Obtaining query information of the cluster based on the OLAP component;

[0248] Determining whether the query information is a large task based on the query information of the cluster, if yes, retaining the query information, otherwise filtering the query information;

[0249] determine a task type based on the reserved query information, and execute a task instruction according to the task type, wherein the task type comprises a killing task and an alarm task;

[0250] If the task type is the killing task, the task instruction is to connect the cluster corresponding to the query information according to the interface information contained in the query information, and to call a preset killing instruction to kill;

[0251] If the task type is the alarm task, the task instruction is to save the query information to a metric interface, and to collect the query information by a monitoring alarm system and perform alarm processing.

[0252] The embodiment of the present application provides a kind of non-volatile computer readable storage medium, computer readable storage medium stores computer executable instruction, the computer executable instruction is executed when one or more processors, implement any one of the kind of implementation of the kind of task killing method based on OLAP component described in various embodiments.

[0253] By way of example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), as external cache memory for a processor. By way of illustration, and not limitation, RAM can be available at many forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memory of the operating environment described herein are intended to include one or any other suitable type of memory in this and / or any other suitable type of memory.

[0254] In the embodiments of the present application, a large task killing method, system, device and medium based on an OLAP component are provided. Query information of a cluster is obtained through the OLAP component, and a large task is analyzed and judged according to the query information, so that an execution task type of the OLAP component corresponding to the cluster containing the large task in the query information is executed, the query performance, availability and stability of the cluster and the business are ensured, in the case of multiple clusters and multiple components, the large task in the cluster can be automatically killed or alarmed, and a unified configuration mode is more convenient for users to manage the OLAP component and the cluster, reduces the operation and maintenance complexity and cost, and solves the technical problems of the traditional large task killing system, which is often designed for specific clusters or components, lacks unified configuration and management mode, increases the operation and maintenance complexity, improves the error risk, and is low in artificial configuration efficiency and high in cost.

[0255] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0256] The terms "first", "second", "third", "fourth" and the like in the description of the present application and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0257] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0258] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0259] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0260] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0261] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various program code storage media.

[0262] The above-described and above-embodied examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0263] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use.

Claims

1. A method for killing a large task based on an OLAP component, characterized in that, The method comprises the following steps: S1, acquiring collection configuration information of different OLAP components, generating cluster query instructions of each of the OLAP components according to the collection configuration information, sending the generated cluster query instructions to each of the OLAP components, executing cluster query by each of the OLAP components, and returning query information of the cluster, wherein the query information comprises a value of SQL execution time consumption, a value of cumulative memory usage, and a value of current memory usage; S2, parsing the query information of the cluster according to the component types of the different OLAP components, calling a judgment instruction corresponding to the component type according to the parsed query information and the component type of the OLAP component to determine whether the query information is a large task, retaining the query information when the value of SQL execution time consumption, the value of cumulative memory usage, and the value of current memory usage all exceed corresponding threshold values, otherwise filtering the query information, and dynamically adjusting the threshold values by using historical data and a machine learning algorithm to optimize the determination criteria of the large task and automatically adjusting the determination criteria according to system load and performance; S3, determining a task type based on the retained query information, and executing a task instruction according to the task type, wherein the task type comprises a killing task and an alarm task; S4, if the task type is the killing task, the task instruction is to connect the cluster corresponding to the query information according to interface information contained in the query information, and to call a preset killing instruction to perform killing; S5, if the task type is the alarm task, the task instruction is to save the query information to a metric interface, and to collect the query information by a monitoring alarm system and perform alarm processing; Before step S2, the method further comprises the following steps: combining each of the OLAP components with a large-scale parallel processing computing framework, generating a query plan according to data distribution of a cluster corresponding to the OLAP component based on a query optimizer of the large-scale parallel processing computing framework, decomposing, by the OLAP component, a received cluster query instruction into a plurality of subtasks based on the large-scale parallel processing computing framework, and executing the subtasks in parallel on a plurality of nodes based on the large-scale parallel processing computing framework.

2. The method of claim 1, wherein the OLAP component-based large task killing method is characterized in that, If the task type is the killing task, the task instruction is to connect the cluster corresponding to the query information according to interface information contained in the query information, and to call a preset killing instruction to perform killing, which further comprises the following steps: saving the query information to a metric interface; collecting the query information from the metric interface by a monitoring alarm system and performing alarm processing.

3. The method of claim 2, wherein the OLAP component-based large task killing method further comprises: The monitoring alarm system collects the query information from the metric interface and performs alarm processing, which specifically comprises the following steps: defining a preset alarm rule based on a key performance indicator of the query information and a preset threshold value; collecting the query information from the metric interface in real time; triggering an alarm when it is detected that the key performance indicator of the query information exceeds the preset threshold value; performing alarm response processing.

4. A large task killing system based on OLAP components, characterized in that, The method comprises the following steps: acquiring a collection configuration information of different OLAP components, generating cluster query instructions of each of the OLAP components according to the collection configuration information, sending the generated cluster query instructions to each of the OLAP components, executing cluster query by each of the OLAP components, and returning query information of the cluster, wherein the query information comprises a value of SQL execution time consumption, a value of cumulative memory usage, and a value of current memory usage; The collection module is configured to acquire collection configuration information of different OLAP components, generate cluster query instructions of each of the OLAP components according to the collection configuration information, send the generated cluster query instructions to each of the OLAP components, execute cluster query by each of the OLAP components, return query information of the cluster, and send the query information to the analysis module, where the query information includes a value of SQL execution time consumption, a value of cumulative memory usage, and a value of current memory usage. The analysis module is configured to analyze the query information of the cluster according to component types of the different OLAP components, call a judgment instruction corresponding to the component type according to the analyzed query information and the component type of the OLAP component to determine whether the query information is a large task, retain the query information when the value of SQL execution time consumption, the value of cumulative memory usage, and the value of current memory usage exceed corresponding threshold values, respectively, otherwise filter the query information, and dynamically adjust threshold values by using historical data and a machine learning algorithm to optimize determination criteria of the large task and automatically adjust the determination criteria according to system load and performance. The configuration module is configured to determine a task type based on the query information, connect the cluster corresponding to the query information according to interface information included in the query information when the task type is a killing task, and call a preset killing instruction to perform killing, save the query information to a metric interface when the task type is an alarm task, and collect the query information by a monitoring alarm module and perform alarm processing. The collection module is further configured to combine each of the OLAP components with a large-scale parallel processing computing framework, generate a query plan according to data distribution of a cluster corresponding to the OLAP component based on a query optimizer of the large-scale parallel processing computing framework, and decompose a received cluster query instruction into a plurality of subtasks based on the large-scale parallel processing computing framework, and execute the subtasks in parallel on a plurality of nodes based on the large-scale parallel processing computing framework.

5. A large task killing device based on OLAP component, characterized in that, The device includes a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the OLAP component-based large task killing method according to instructions in the program code.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is configured to execute the OLAP component-based large task killing method.

Images