Large task searching and killing method, system and device based on OLAP component and medium

Abstract

The invention discloses a large task searching and killing method, system and device based on an OLAP component and a medium, query information of a cluster is obtained through the OLAP component, and a large task is analyzed and judged according to the query information, so that the execution task type of the large task contained in the cluster corresponding to the OLAP component in the query information is executed, the query performance of the cluster and a service is guaranteed, and the efficiency of searching and killing the large task is improved. Availability and stability are realized; under the condition of multiple clusters and multiple components, large tasks in the clusters can be automatically searched and killed or alarmed, a uniform configuration mode is more convenient for a user to manage OLAP components and clusters, the operation and maintenance complexity and cost are reduced, and the problem that a traditional large task searching and killing system is often designed for specific clusters or components, so that the large task searching and killing system is inconvenient to use is solved. The technical problems that the operation and maintenance complexity is increased, the error risk is also improved, the manual configuration efficiency is low and the cost is high due to the lack of a unified configuration and management mode in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a large task detection and killing method, system, device and medium based on OLAP components. Background Art

[0002] OLAP (On-line Analytical Processing) is a key data analysis technology that allows users to quickly analyze and summarize data from multiple perspectives. This technology is particularly suitable for processing and analyzing large amounts of data, such as information in enterprise data warehouses. OLAP systems typically support complex queries, including operations such as slicing, dicing, summarizing, and comparing multidimensional data.

[0003] With the advent of the big data era, the amount of data that organizations need to process has increased dramatically, which requires more powerful OLAP systems to meet analysis needs. Modern OLAP components such as ClickHouse, Presto, and Doris have emerged. They provide high-performance data processing capabilities, support real-time analysis and fast query, and meet the needs of different business scenarios.

[0004] However, as the scale of OLAP clusters increases and businesses diversify, a single large task may have a significant impact on the performance of the entire cluster, leading to resource shortages and even unstable services. In this case, a large task detection system is particularly important. This system can monitor the load of the cluster and automatically take measures when a large task is detected, such as limiting resource usage, delaying execution, or terminating the task to protect the stability and availability of the cluster.

[0005] Traditional large-task killing systems are often designed for specific clusters or components, lacking a unified configuration and management method. This not only increases the complexity of operation and maintenance, but also increases the risk of errors. In a multi-component, multi-cluster environment, operation and maintenance personnel need to manage a large number of configurations, which is not only inefficient but also costly. Summary of the invention

[0006] The present application provides a large task killing method, system, device and medium based on OLAP components, which solves the technical problems that traditional large task killing systems are often designed for specific clusters or components, lack unified configuration and management methods, increase the complexity of operation and maintenance, increase the risk of errors, and have low efficiency and high cost of manual configuration.

[0007] In view of this, the first aspect of the present application provides a large task killing method based on OLAP components, the method comprising:

[0008] Obtain cluster query information based on OLAP components;

[0009] Determine whether the query information is a large task based on the query information of the cluster, and if so, retain the query information, otherwise filter the query information;

[0010] Determine the task type based on the retained query information, and execute the task instruction according to the task type, wherein the task type includes a check and kill task and an alarm task;

[0011] If the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset killing instruction to perform the killing;

[0012] If the task type is an alarm task, the task instruction is to save the query information to a measurement interface, and the monitoring alarm system collects the query information and performs alarm processing.

[0013] Optionally, the acquiring query information of the cluster based on the OLAP component specifically includes:

[0014] Get the collection configuration information of different OLAP components;

[0015] Generate a cluster query instruction for each of the OLAP components according to the acquisition configuration information;

[0016] The generated cluster query instruction is sent to each of the OLAP components, and each of the OLAP components executes the cluster query and returns cluster query information.

[0017] Optionally, the acquiring query information of the cluster based on the OLAP component further includes:

[0018] Combining each of the OLAP components with a large-scale parallel processing computing framework;

[0019] A query optimizer based on the large-scale parallel processing computing framework generates a query plan according to the data distribution of the cluster corresponding to the OLAP component.

[0020] Optionally, after combining each of the OLAP components with a large-scale parallel processing computing framework, the method further includes:

[0021] Based on the large-scale parallel processing computing framework, the OLAP component decomposes the received cluster query instruction into a plurality of subtasks;

[0022] Based on the large-scale parallel processing computing framework, the subtasks are executed in parallel on a plurality of nodes.

[0023] Optionally, judging whether the query information is a large task based on the query information of the cluster, and if so, retaining the query information, and otherwise filtering the query information specifically includes:

[0024] parsing the query information of the cluster according to component types of different OLAP components;

[0025] According to the parsed query information and in combination with the component type of the OLAP component, a judgment instruction corresponding to the component type is called to judge whether the query information is a large task. If so, the query information is retained; otherwise, the query information is filtered.

[0026] Optionally, if the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information included in the query information, and call a preset killing instruction to perform the killing, and further includes:

[0027] Saving the query information to a measurement interface;

[0028] The monitoring and alarm system collects the query information from the measurement interface and performs alarm processing.

[0029] Optionally, the monitoring alarm system collects the query information from the metric interface and performs alarm processing specifically including:

[0030] Defining preset alarm rules, wherein the preset alarm rules are key performance indicators and preset thresholds based on the query information;

[0031] collecting the query information from the measurement interface in real time;

[0032] When it is detected that the key performance indicator of the query information exceeds the preset threshold, an alarm is triggered;

[0033] Perform alarm response processing.

[0034] A second aspect of the present application provides a large task killing system based on an OLAP component, the system comprising:

[0035] Acquisition module, analysis module and configuration module;

[0036] The acquisition module is used to obtain query information of the cluster based on the OLAP component and send the query information to the analysis module;

[0037] The analysis module is used to determine whether the query information is a large task based on the query information of the cluster, and if so, retain the query information; otherwise, filter the query information and send the retained query information to the configuration module;

[0038] The configuration module is used to determine the task type based on the query information. If the task type is a detection and killing task, the cluster corresponding to the query information is connected according to the interface information contained in the query information, and the preset detection and killing instruction is called to perform the detection and killing. If the task type is an alarm task, the query information is saved to the measurement interface, and the monitoring and alarm module collects the query information and performs alarm processing.

[0039] Optionally, the acquisition module is specifically used for:

[0040] Get the collection configuration information of different OLAP components;

[0041] Generate a cluster query instruction for each of the OLAP components according to the acquisition configuration information;

[0042] The generated cluster query instruction is sent to each of the OLAP components, and each of the OLAP components executes the cluster query and returns cluster query information.

[0043] Optionally, the acquisition module is further used for:

[0044] Combining each of the OLAP components with a large-scale parallel processing computing framework;

[0045] A query optimizer based on the large-scale parallel processing computing framework generates a query plan according to the data distribution of the cluster corresponding to the OLAP component.

[0046] Optionally, the acquisition module is further used for:

[0047] Based on the large-scale parallel processing computing framework, the OLAP component decomposes the received cluster query instruction into a plurality of subtasks;

[0048] Based on the large-scale parallel processing computing framework, the subtasks are executed in parallel on a plurality of nodes.

[0049] Optionally, the analysis module is specifically used for:

[0050] parsing the query information of the cluster according to component types of different OLAP components;

[0051] According to the parsed query information and in combination with the component type of the OLAP component, a judgment instruction corresponding to the component type is called to judge whether the query information is a large task. If so, the query information is retained; otherwise, the query information is filtered.

[0052] Optionally, it also includes:

[0053] The monitoring and alarm module is used to collect the query information stored in the measurement interface and perform alarm processing.

[0054] Optionally, the monitoring alarm system is specifically used for:

[0055] Defining preset alarm rules, wherein the preset alarm rules are key performance indicators and preset thresholds based on the query information;

[0056] collecting the query information from the measurement interface in real time;

[0057] When it is detected that the key performance indicator of the query information exceeds the preset threshold, an alarm is triggered;

[0058] Perform alarm response processing.

[0059] A third aspect of the present application provides a large task killing device based on an OLAP component, the device comprising a processor and a memory:

[0060] The memory is used to store program codes and transmit the program codes to the processor;

[0061] The processor is used to execute the steps of the method for killing large tasks based on OLAP components as described in the first aspect according to the instructions in the program code.

[0062] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0063] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0064] In the present application, a large task detection and killing method, system, device and medium based on OLAP component are provided, which obtains query information of the cluster through the OLAP component, and judges the large task according to the query information, so as to execute the execution task type of the large task contained in the cluster corresponding to the OLAP component in the query information, thereby ensuring the query performance, availability and stability of the cluster and the business; in the case of multiple clusters and multiple components, the large tasks in the cluster can be automatically detected and killed or alarmed, and the unified configuration mode is more convenient for users to manage OLAP components and clusters, reducing the complexity and cost of operation and maintenance, and solving the technical problems that the traditional large task detection and killing system is often designed for specific clusters or components, lacks a unified configuration and management method, which increases the complexity of operation and maintenance, increases the risk of errors, and has low efficiency and high cost of manual configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1This is a method flow chart of a large task killing method based on OLAP components in an embodiment of the present application;

[0066] Figure 2 This is a system architecture diagram of a large task killing system based on OLAP components in an embodiment of the present application;

[0067] Figure 3 It is a structural diagram of a large task killing device based on OLAP components in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] The present application designs a large task killing method, system, device and medium based on OLAP components, which solves the technical problems that traditional large task killing systems are often designed for specific clusters or components, lack unified configuration and management methods, increase the complexity of operation and maintenance, increase the risk of errors, and have low efficiency and high cost of manual configuration.

[0070] For easier understanding, see Figure 1 , Figure 1 Flow chart of the method for killing large tasks based on OLAP components in the embodiment of the present application. Figure 1 As shown, specifically:

[0071] S1. Obtaining cluster query information based on OLAP components;

[0072] Specifically include:

[0073] Get the collection configuration information of different OLAP components;

[0074] Generate cluster query instructions for each OLAP component based on the collected configuration information;

[0075] The generated cluster query instruction is sent to each OLAP component, and each OLAP component executes the cluster query and returns the cluster query information.

[0076] It should be noted that the list of OLAP components that need to be monitored should be determined, such as ClickHouse, Presto, Doris, Druid, etc.

[0077] Configure data collection tools or agents for each component to collect query information for the cluster.

[0078] Ensure that the collection tool can obtain key information including query statements, execution time, resource usage, etc.

[0079] The overall logic of different components is consistent, and the implementation methods are adjusted according to different components. For example, in the logic of obtaining key information of the cluster, presto calls the interface http: / / host:9999 / v1 / query to obtain query information, while doris obtains it by calling the interface http: / / host:8031 / rest / v2 / manager / query / current_queries.

[0080] This application uses Presto as an example to illustrate an application example. According to the collection configuration information, for example, by configuring action.presto_query_kill.analyze_period in milliseconds, the time interval for calling the Presto interface is configured.

[0081] Taking 10 seconds as an example, the Presto interface http: / / host:9999 / v1 / query is called every 10 seconds to obtain cluster query information. The query information includes SQL statements, SQL execution time, current memory usage, cumulative memory usage, query ID and other information.

[0082] Each different OLAP component can obtain different query information, as shown in the following table:

[0083]

[0084] Furthermore, obtaining query information of the cluster based on the OLAP component also includes:

[0085] Combine various OLAP components with a large-scale parallel processing computing framework;

[0086] The query optimizer based on the massively parallel processing computing framework generates a query plan according to the data distribution of the corresponding cluster of the OLAP component.

[0087] Furthermore, after combining each OLAP component with a large-scale parallel processing computing framework, the following is also included:

[0088] Based on the large-scale parallel processing computing framework, the OLAP component decomposes the received cluster query instructions into several subtasks;

[0089] Based on a large-scale parallel processing computing framework, subtasks are executed in parallel on several nodes.

[0090] It should be noted that combining OLAP components with MPP (massively parallel processing) computing framework can improve the query performance of OLAP systems, especially when processing large-scale data sets. Specifically, OLAP components can be integrated into the MPP computing framework so that query operations can be executed in parallel on multiple computing nodes.

[0091] The query optimizer analyzes the query request and generates an efficient query execution plan based on the distribution of data in the cluster. This plan determines how to decompose the query, how to transfer data, and how to allocate computing resources.

[0092] The OLAP component will decompose the query into multiple subtasks according to the query plan, and each subtask can be executed independently. This decomposition can be based on data partitioning, sharding or other logic.

[0093] In the MPP computing framework, subtasks are assigned to different nodes in the cluster. Each node executes its assigned subtasks independently and simultaneously, which can significantly reduce the overall query processing time.

[0094] Suppose we have a large data warehouse stored in an MPP computing framework consisting of 100 nodes. We want to perform a complex analytical query, such as calculating the annual sales total for each region.

[0095] Query reception: Users submit query requests through OLAP front-end tools.

[0096] Query optimization: The query optimizer in the MPP computing framework analyzes the query and generates a query plan taking into account the distribution of data on various nodes.

[0097] Task decomposition: The query is decomposed into multiple subtasks. For example, the sales data for each region is calculated on different nodes.

[0098] Parallel execution: Each node processes its assigned subtask in parallel to calculate the annual sales totals for their respective regions.

[0099] Result aggregation: After each node completes the calculation, it sends the result back to the coordination node. The coordination node is responsible for merging the results of all subtasks to form the final query result.

[0100] Result return: The coordination node returns the aggregated result to the user.

[0101] Furthermore, automated configuration management tools can be used to simplify the collection configuration of different components. Standardized query information formats can also be used to facilitate subsequent processing and analysis.

[0102] Furthermore, support for new OLAP components can be easily extended using plug-in or adapter patterns without modifying the existing code base.

[0103] Plugin or adapter pattern definition:

[0104] Plugin mode: allows the system to dynamically load and unload modules at runtime. These modules implement specific interfaces to provide additional functionality.

[0105] Adapter pattern: used to ensure that different OLAP components can be integrated with the monitoring system in a unified way, even if their data access interfaces and data formats may be different.

[0106] Designing a unified interface:

[0107] Define a common data collection interface that all plug-ins or adapters must implement. In this way, regardless of the specific implementation of the OLAP component, the monitoring system can interact with it in a unified way.

[0108] Development of specific adapters:

[0109] Specific adapters are developed for each OLAP component. These adapters implement a unified interface, but the internal logic is optimized for specific components to ensure efficient data collection.

[0110] No need to modify existing code:

[0111] Through the plug-in or adapter mode, when adding support for new OLAP components, there is no need to modify the existing code of the monitoring system. This reduces maintenance costs and improves development efficiency.

[0112] Easy to expand and maintain:

[0113] When a new OLAP component needs to be supported, only a new adapter or plug-in needs to be developed and integrated into the system without large-scale modifications to the existing code.

[0114] Furthermore, a load balancing mechanism for data collection is introduced to ensure that data collection will not become a bottleneck for OLAP components under high concurrency conditions.

[0115] In high-concurrency situations, ensure that data collection operations do not negatively impact the performance of OLAP components and avoid becoming a system bottleneck.

[0116] Implementation strategy:

[0117] Distributed collection: Data collection tasks are distributed to multiple collection nodes, with each node responsible for a part of the data collection work, thereby dispersing the load.

[0118] Sampling and aggregation: During the data collection process, sampling and aggregation techniques are used to reduce the amount of data that needs to be transmitted and processed.

[0119] Priority scheduling: Set different priorities for data collection tasks based on the importance and real-time requirements of the data to ensure that key data is collected first.

[0120] Monitor and adjust:

[0121] Monitor the load and performance of data collection in real time, and dynamically adjust collection strategies and resource allocation based on actual conditions.

[0122] Failover and Redundancy:

[0123] Implement failover and redundancy mechanisms between collection nodes to ensure high availability and reliability of data collection.

[0124] Resource optimization:

[0125] Optimize resource allocation for data collection, such as CPU, memory, and network bandwidth, based on resource usage and performance indicators of OLAP components.

[0126] S2. judging whether the query information is a large task based on the query information of the cluster, and if so, retaining the query information, otherwise filtering the query information;

[0127] Specifically include:

[0128] Parse the query information of the cluster according to the component types of different OLAP components;

[0129] According to the parsed query information and the component type of the OLAP component, a judgment instruction corresponding to the component type is called to judge whether the query information is a large task. If so, the query information is retained, otherwise the query information is filtered.

[0130] It should be noted that the collected query information is parsed to extract key indicators such as execution time, memory usage, etc. According to the preset threshold or rule, it is judged whether the query meets the conditions of the large task. For large tasks that meet the conditions, their query information is retained; those that do not meet the conditions are filtered out.

[0131] This application uses Presto as an example to illustrate an application example. Taking 10 seconds as an example, the collected Presto query information is obtained every 10 seconds. The query information includes the value of SQL execution time, the value of accumulated memory used, and the value of current memory used;

[0132] Whether it is a large task based on the query information can be determined by retaining the query information if the SQL execution time, the accumulated memory used, and the current memory used exceed corresponding thresholds; otherwise, the query information is filtered.

[0133] Furthermore, based on intelligent analysis tools, historical data and machine learning algorithms can be used to optimize the judgment criteria for large tasks, realize dynamic threshold adjustment, and automatically adjust the judgment criteria according to system load and performance.

[0134] S3, determining the task type based on the retained query information, and executing the task instruction according to the task type, wherein the task type includes a detection and killing task and an alarm task;

[0135] It should be noted that the retained query information is classified to determine its task type, such as a detection task or an alarm task. According to the task type, a corresponding processing strategy and instruction are formulated.

[0136] In the monitoring system of the OLAP component, determining the task type based on the retained query information and executing the corresponding task instructions is a key process. This process involves analyzing the query information, determining the task type, and executing the corresponding operation. The following is a detailed explanation:

[0137] Analysis query information:

[0138] The monitoring system first collects and analyzes query information from the OLAP component. This information may include query execution time, resource consumption (such as CPU, memory usage), query source, query complexity, etc.

[0139] Determine the task type:

[0140] Based on the analysis results, the system needs to determine whether each query information triggers the preset task type conditions. Task types usually include:

[0141] Killing tasks: When the resources consumed by a query exceed the preset threshold, or the query execution time is too long, which may have a negative impact on system performance, the system marks it as a killing task.

[0142] Alarm task: When a query exhibits abnormal behavior but has not yet reached the point where it requires immediate detection, the system may mark it as an alarm task to alert the administrator to conduct further inspection.

[0143] Furthermore, a more flexible task type definition mechanism can be designed to adapt to different business needs and scenarios, realize automatic identification and classification of task types, and reduce manual intervention. Specifically:

[0144] Design a more flexible task type definition mechanism to enable the monitoring system to automatically adapt to different business needs and scenarios, and reduce manual intervention by automatically identifying and classifying task types. The following are the specific steps and methods to achieve this goal:

[0145] Define the task characteristics:

[0146] Determine the key characteristics and indicators of different task types, such as query execution time, resource consumption, query frequency, etc. These characteristics will be used to distinguish between killing tasks and alarm tasks.

[0147] Using machine learning algorithms:

[0148] Using machine learning algorithms, such as classification algorithms (decision trees, random forests, support vector machines, etc.), models are trained based on historical data to automatically identify and classify task types.

[0149] Through continuous learning and optimization, the model can adapt to new business scenarios and changes and improve the accuracy of task classification.

[0150] Real-time monitoring and analysis:

[0151] Implement a real-time monitoring system to continuously collect query information and performance indicators for timely analysis and processing.

[0152] Use stream processing technologies (such as Apache Kafka, Apache Flink, etc.) to process real-time data streams and improve data processing efficiency and response speed.

[0153] Dynamic Threshold Adjustment:

[0154] Set dynamic thresholds that automatically adjust based on system load, historical data, and real-time performance indicators. This helps maintain reasonable monitoring sensitivity in different business scenarios.

[0155] Dynamic thresholds can be achieved through machine learning model predictions or rule-based adaptive algorithms.

[0156] Automatic identification of task types:

[0157] Based on predefined features and dynamic thresholds, the monitoring system automatically determines whether the query information meets the conditions of the detection task or the alarm task.

[0158] The system can set multiple levels of thresholds to distinguish tasks of different severity, such as mild warnings, moderate warnings, and severe cleaning.

[0159] Task classification and execution:

[0160] Once the task type is automatically identified, the system will execute the corresponding task instructions according to the task type. For example, for a kill task, the system will terminate or limit queries that consume too much resources; for an alarm task, the system will record the event and notify the administrator.

[0161] The system can set multiple levels of task instructions to accommodate tasks of different severity.

[0162] This application uses Presto as an example to illustrate an application example. Taking 10 seconds as an example, the retained query information is obtained every 10 seconds, and the task type is determined according to the action.presto_query_kill.type configured in the query information. When action.presto_query_kill.type is find_and_kill, it is a kill task, and when action.presto_query_kill.type is find_and_warn, it is an alarm task.

[0163] S4. If the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset killing instruction to perform the killing;

[0164] It should be noted that when the task type is determined as "killing task", it means that the monitoring system has identified that certain query operations may have a negative impact on the performance of OLAP components, such as occupying a large number of resources for a long time or causing system instability. In this case, the system needs to take measures to terminate these queries to restore system performance and stability. The following are the specific execution steps of "killing task":

[0165] Get interface information:

[0166] Extract interface information from the query information collected by the monitoring system. This information usually includes database connection details such as host address, port number, authentication information, etc., which are required to connect to the OLAP component cluster.

[0167] Establishing a connection:

[0168] Using the extracted interface information, the monitoring system establishes a connection with the OLAP component cluster through the corresponding client or API. This step ensures that the monitoring system can interact directly with the target cluster.

[0169] Call the killing command:

[0170] Once the connection is established, the monitoring system will call the preset kill command. The kill command is a command designed to terminate or limit query operations that are identified as excessive resource consumption.

[0171] For example, in Presto, you can use the KILL QUERY command to terminate an executing query; in ClickHouse, you can use KILL QUERYWHERE query_id='xxxx' to terminate a query with a specific ID.

[0172] Execute the killing:

[0173] The system executes a kill command, which will cause the target query to be terminated. This process may involve sending a kill signal to the query process, or operating through the cluster management interface.

[0174] Confirm the killing results:

[0175] After the killing operation is completed, the monitoring system needs to confirm whether the killing is successful. This can be achieved by checking the query status, resource release status or query execution log.

[0176] Records and Notifications:

[0177] Regardless of whether the killing operation is successful, the monitoring system should record the event, including the reason, time, operation performed, and result of the killing. In addition, the system should notify the relevant operation and maintenance team or system administrator so that they can take further actions, such as analyzing the cause of the killing and optimizing the query.

[0178] Resource recovery:

[0179] After the killing operation, the system should ensure that the previously occupied resources (such as memory, CPU time, etc.) are properly reclaimed so that other queries can utilize these resources.

[0180] Audit and Compliance:

[0181] In some cases, killing operations may need to comply with specific audit and compliance requirements. The monitoring system should ensure that killing operations are performed in accordance with these requirements and keep corresponding records for audit purposes.

[0182] In this way, killing tasks helps maintain the performance and stability of OLAP components and prevents a single query from over-consuming resources and affecting the operation of the entire system.

[0183] S5. If the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset killing instruction to kill, and then further includes:

[0184] Save the query information to the measurement interface;

[0185] The monitoring and alarm system collects query information from the measurement interface and performs alarm processing.

[0186] It should be noted that for the killing task, the corresponding cluster is connected through the interface information in the query information, and the preset killing instruction is called to terminate the large task or limit the resources.

[0187] Furthermore, the monitoring alarm system collects query information from the metric interface and performs alarm processing, specifically including:

[0188] Define preset alarm rules, which are key performance indicators and preset thresholds based on query information;

[0189] Collect query information from the measurement interface in real time;

[0190] When it is detected that the key performance indicators of the query information exceed the preset threshold, an alarm is triggered;

[0191] Perform alarm response processing.

[0192] It should be noted that a set of alarm rules are defined, which are based on the key performance indicators (KPIs) of the query information and preset thresholds;

[0193] Monitor query information in real time and trigger an alarm when any indicator is detected to exceed the preset threshold;

[0194] When an alarm is triggered, the system automatically executes a series of predefined response measures, such as sending a notification to the system administrator, recording an event log, launching an automatic diagnostic program, or performing an automatic repair operation;

[0195] During the alarm processing, the system can take different levels of response measures according to the severity and urgency level of the alarm;

[0196] The system supports custom alarm notification methods, including but not limited to email, SMS, application push notification, etc., to ensure that relevant personnel are notified in a timely manner;

[0197] Provides a user interface that enables system administrators to view alarm history, current alarm status, and configure and manage alarm rules.

[0198] Customizing monitoring strategies based on the characteristics of OLAP components is a complex but critical process that can help ensure system performance and stability. The following are some key steps and considerations for customizing effective monitoring strategies for different OLAP components:

[0199] Understanding component features:

[0200] First, you need to have a deep understanding of the architecture, performance characteristics, common query patterns, and resource consumption characteristics of each OLAP component. For example, some components may be better at processing real-time queries, while others may be more suitable for batch processing.

[0201] Define Key Performance Indicators (KPIs):

[0202] Determine which metrics are critical to monitor. Common KPIs include query response time, resource utilization (CPU, memory, disk I / O), number of concurrent queries, query throughput, etc.

[0203] To configure data collection:

[0204] Configure data collection tools based on the characteristics of the components to ensure that all key performance data can be collected. This may include configuring API calls, log file parsing, or using specific monitoring tools.

[0205] To set thresholds and alerts:

[0206] Set thresholds for each key metric and trigger alerts when the metric exceeds these thresholds. The thresholds should be set based on the component's performance baseline and business requirements.

[0207] Consider the query pattern:

[0208] Analyze the query patterns of components and identify which queries are common and which are abnormal. This helps to set more precise monitoring rules, such as setting different monitoring strategies for specific types of queries.

[0209] To achieve dynamic adjustment:

[0210] Leverage machine learning or other automation tools to dynamically adjust monitoring policies, for example, automatically adjusting thresholds based on historical data and real-time performance metrics.

[0211] Integrated Massively Parallel Processing (MPP) features:

[0212] If the OLAP component is an MPP architecture, the monitoring strategy should be able to reflect the distributed nature of the cluster, for example, monitoring data distribution, load balancing between nodes, and query execution across nodes.

[0213] Consider data security and privacy:

[0214] During the monitoring process, ensure compliance with data security and privacy regulations. For sensitive data, encryption or anonymization may be required.

[0215] User customization and extensibility:

[0216] Provides the ability for users to customize monitoring rules, allowing users to adjust monitoring strategies according to their specific needs.

[0217] Tested and validated:

[0218] Before implementing a monitoring strategy, perform thorough testing and validation to ensure that the monitoring tools and strategy can correctly capture and report performance issues.

[0219] Documentation and Training:

[0220] Provide detailed documentation for monitoring strategies and provide necessary training to operations teams to ensure they understand monitoring tools and strategies.

[0221] Continuous Optimization:

[0222] Monitoring strategies are not static and need to be continuously optimized and adjusted based on system changes and business needs.

[0223] Through the above steps, you can customize effective monitoring strategies for different OLAP components to help timely discover and solve performance problems and improve system reliability and efficiency.

[0224] Furthermore, the killing strategy can be dynamically adjusted according to the task characteristics and cluster status.

[0225] Furthermore, priority and queue mechanisms are designed for different types of tasks to ensure that critical tasks are processed first.

[0226] Furthermore, a rollback mechanism is provided for the killing task so that a quick recovery can be achieved when a problem is caused by the killing operation.

[0227] If the task type is an alarm task, the task instruction is to save the query information to the measurement interface, and the monitoring alarm system collects the query information and performs alarm processing.

[0228] It should be noted that for alarm tasks, the query information is saved to the measurement interface. The monitoring alarm system collects and processes alarms based on this information.

[0229] Integrate multiple alert channels, such as email, SMS, real-time messaging, etc., to ensure that relevant personnel are notified in a timely manner.

[0230] Implement automated analysis of alarm information and response suggestions to help quickly locate problems and take action.

[0231] In addition, you can develop custom alarm rules and triggers to meet different monitoring needs. At the same time, you can also provide visual display of alarm information, such as dashboards and alarm boards, to achieve visual display and historical analysis of alarm information, helping the operation and maintenance team to better understand the system status and trends.

[0232] This application uses Presto as an example to illustrate an application example. Taking 10 seconds as an example, the large task query information of Presto is obtained every 10 seconds.

[0233] When action.presto_query_kill.type configured in the query information is equal to find_and_kill, the Presto cluster is connected according to the presto_jdbc information configured in the query information and the allsystem.runtime.kill_query() command is called to kill the corresponding large task query information.

[0234] When action.presto_query_kill.type configured in the query information is equal to find_and_warn, no killing process is performed.

[0235] Regardless of whether the query information is processed or not, the query information will be output to the measurement interface, and the preset time length, such as 120 seconds, will be saved in the measurement interface. The query information is collected by the monitoring and alarm system, and the monitoring and alarm system will send email alarms, phone alarms, SMS alarms, etc. to the corresponding users according to the cluster corresponding to the query information.

[0236] See also Figure 2 , Figure 2 : is a system architecture diagram of a large task killing system based on OLAP components in an embodiment of the present application, such as Figure 2 As shown, specifically:

[0237] Collection module 201, analysis module 202 and configuration module 203;

[0238] The acquisition module 201 is used to obtain query information of the cluster based on the OLAP component and send the query information to the analysis module 202;

[0239] The analysis module 202 is used to determine whether the query information is a large task based on the query information of the cluster, and if so, retain the query information, otherwise filter the query information, and send the retained query information to the configuration module 203;

[0240] The configuration module 203 is used to determine the task type based on the query information. If the task type is a detection and killing task, the cluster corresponding to the query information is connected according to the interface information contained in the query information, and the preset detection and killing instructions are called for detection and killing. If the task type is an alarm task, the query information is saved to the measurement interface, and the monitoring and alarm module collects the query information and performs alarm processing.

[0241] Another embodiment of the present invention provides a large task killing device based on OLAP components, such as Figure 3 As shown, the device 10 includes:

[0242] One or more processors 110 and memory 120, Figure 3A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0243] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0244] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for constructing a multilingual phoneme representation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for constructing a multilingual phoneme representation model in the above method embodiment.

[0245] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0246] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0247] Obtain cluster query information based on OLAP components;

[0248] Based on the query information of the cluster, determine whether the query information is a large task. If so, retain the query information; otherwise, filter the query information;

[0249] Determine the task type based on the retained query information, and execute the task instruction according to the task type, wherein the task type includes a detection and killing task and an alarm task;

[0250] If the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset killing instruction to perform the killing;

[0251] If the task type is an alarm task, the task instruction is to save the query information to the measurement interface, and the monitoring alarm system collects the query information and performs alarm processing.

[0252] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, any implementation of the large task killing method based on OLAP components described in the above embodiments is implemented.

[0253] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0254] In the embodiment of the present application, a large task detection and killing method, system, device and medium based on OLAP component are provided, which obtains query information of the cluster through the OLAP component, and judges the large task according to the query information, so as to execute the execution task type of the large task contained in the cluster corresponding to the OLAP component in the query information, thereby ensuring the query performance, availability and stability of the cluster and the business; in the case of multiple clusters and multiple components, the large tasks in the cluster can be automatically detected and killed or alarmed, and the unified configuration mode is more convenient for users to manage OLAP components and clusters, reducing the complexity and cost of operation and maintenance, and solving the technical problems that the traditional large task detection and killing system is often designed for specific clusters or components, lacks a unified configuration and management method, which increases the complexity of operation and maintenance, increases the risk of errors, and has low efficiency and high cost of manual configuration.

[0255] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0256] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0257] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0259] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0262] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0263] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A large task killing method based on OLAP components, characterized in that: include: Obtain cluster query information based on OLAP components; Determine whether the query information is a large task based on the query information of the cluster, and if so, retain the query information, otherwise filter the query information; Determine the task type based on the retained query information, and execute the task instruction according to the task type, wherein the task type includes a check and kill task and an alarm task; If the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset killing instruction to perform the killing; If the task type is an alarm task, the task instruction is to save the query information to a measurement interface, and the monitoring alarm system collects the query information and performs alarm processing.

2. The large task killing method based on OLAP component according to claim 1 is characterized in that: The acquisition of cluster query information based on the OLAP component specifically includes: Get the collection configuration information of different OLAP components; Generate a cluster query instruction for each of the OLAP components according to the acquisition configuration information; The generated cluster query instruction is sent to each of the OLAP components, and each of the OLAP components executes the cluster query and returns cluster query information.

3. The large task detection and killing method based on OLAP component according to claim 2 is characterized in that: The obtaining of cluster query information based on the OLAP component also includes: Combining each of the OLAP components with a large-scale parallel processing computing framework; A query optimizer based on the large-scale parallel processing computing framework generates a query plan according to the data distribution of the cluster corresponding to the OLAP component.

4. The large task checking and killing method based on OLAP component according to claim 3 is characterized in that: After combining each of the OLAP components with the large-scale parallel processing computing framework, the following further comprises: Based on the large-scale parallel processing computing framework, the OLAP component decomposes the received cluster query instruction into a plurality of subtasks; Based on the large-scale parallel processing computing framework, the subtasks are executed in parallel on a plurality of nodes.

5. The large task killing method based on OLAP component according to claim 2 is characterized in that: The determining whether the query information is a large task based on the query information of the cluster, if so, retaining the query information, otherwise filtering the query information specifically includes: parsing the query information of the cluster according to component types of different OLAP components; According to the parsed query information and in combination with the component type of the OLAP component, a judgment instruction corresponding to the component type is called to judge whether the query information is a large task. If so, the query information is retained; otherwise, the query information is filtered.

6. The large task checking and killing method based on OLAP component according to claim 5 is characterized in that: If the task type is a killing task, the task instruction is to connect to the cluster corresponding to the query information according to the interface information contained in the query information, and call the preset killing instruction to perform killing, and further includes: Saving the query information to a measurement interface; The monitoring and alarm system collects the query information from the measurement interface and performs alarm processing.

7. The large task killing method based on OLAP component according to claim 6 is characterized in that: The monitoring alarm system collects the query information from the measurement interface and performs alarm processing specifically including: Defining preset alarm rules, wherein the preset alarm rules are key performance indicators and preset thresholds based on the query information; collecting the query information from the measurement interface in real time; When it is detected that the key performance indicator of the query information exceeds the preset threshold, an alarm is triggered; Perform alarm response processing.

8. A large task killing system based on OLAP components, characterized in that: include: Acquisition module, analysis module and configuration module; The acquisition module is used to obtain query information of the cluster based on the OLAP component and send the query information to the analysis module; The analysis module is used to determine whether the query information is a large task based on the query information of the cluster, and if so, retain the query information; otherwise, filter the query information and send the retained query information to the configuration module; The configuration module is used to determine the task type based on the query information. If the task type is a detection and killing task, the cluster corresponding to the query information is connected according to the interface information contained in the query information, and the preset detection and killing instruction is called to perform the detection and killing. If the task type is an alarm task, the query information is saved to the measurement interface, and the monitoring and alarm module collects the query information and performs alarm processing.

9. A large task detection and killing device based on OLAP components, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the large task killing method based on OLAP components according to any one of claims 1 to 6 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the large task killing method based on OLAP components according to any one of claims 1 to 6.

Images