A task processing method, apparatus, device, medium and product

Abstract

The application discloses a task processing method and device, equipment, medium and product. The method is applied to a distributed database system, and the distributed database system is used for executing a data processing task. The method comprises the following steps: when a patrol time of the distributed database system is reached, execution information of N first data processing tasks to be patrolled in the distributed database system is acquired, N is a positive integer; statistical analysis is performed on the execution information, so that resource consumption data when each first data processing task is executed is obtained; based on the resource consumption data when each first data processing task is executed, a second data processing task is located from the N first data processing tasks, and a data quantity of resource consumption data when the second data processing task is executed is higher than a preset consumption threshold; and an intervention process of the second data processing task is processed. The application can improve the task manageability of the distributed database system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of database technology, specifically to a task processing method, a task processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] A distributed database system is a system that enables multiple nodes to perform data processing tasks in parallel through collaborative work. During the operation and maintenance of distributed database systems, it is common to encounter situations where the total number of tasks remains relatively constant, but the proportion of large tasks increases, leading to sudden load spikes and compromised customer SLAs (Service Level Agreements). Large tasks refer to data processing tasks that significantly exceed the scope of normal operations in terms of data volume, complexity, or resource consumption. Existing methods struggle to quickly identify large tasks in a distributed database system, resulting in time-consuming problem analysis and difficulties in timely resolution. Summary of the Invention

[0003] This application provides a task processing method, apparatus, device, medium, and product that can improve the task manageability of a distributed database system by accurately locating a second data processing task that consumes a large amount of resources.

[0004] On one hand, embodiments of this application provide a task processing method applied to a distributed database system, which is used to perform data processing tasks; the method includes:

[0005] When the inspection time of the distributed database system is reached, obtain the execution information of the N first data processing tasks to be inspected in the distributed database system, where N is a positive integer;

[0006] Analyze the execution information to obtain the resource consumption data when each first data processing task is executed;

[0007] Based on the resource consumption data when each first data processing task is executed, the second data processing task is located from N first data processing tasks. The amount of resource consumption data when the second data processing task is executed is higher than the preset consumption threshold.

[0008] Intervene in the execution process of the second data processing task.

[0009] On one hand, embodiments of this application provide a task processing apparatus applied to a distributed database system, the distributed database system being used to perform data processing tasks; the apparatus includes:

[0010] The acquisition unit is used to acquire the execution information of the N first data processing tasks to be inspected in the distributed database system when the inspection time of the distributed database system arrives, where N is a positive integer;

[0011] The processing unit is used to analyze the execution information to obtain the resource consumption data when each first data processing task is executed;

[0012] The processing unit is also used to locate the second data processing task from the N first data processing tasks based on the resource consumption data when each first data processing task is executed, wherein the amount of resource consumption data when the second data processing task is executed is higher than a preset consumption threshold.

[0013] The processing unit is also used to intervene in the execution process of the second data processing task.

[0014] In one embodiment, the distributed database system includes multiple data nodes, each data node running a data processing thread pool, and a data processing task in the distributed database system is assigned to the data processing thread pool on at least one data node for execution; the processing unit is further configured to:

[0015] The system retrieves the operating load of each data node by calling the acquisition unit.

[0016] Based on the operating load of each data node, a first data node is determined from multiple data nodes. The operating load of the first data node is higher than a preset load threshold, and a first data processing thread pool is running on the first data node.

[0017] The data processing tasks executed in the first data processing thread pool are identified as the first data processing tasks to be inspected in the distributed database system.

[0018] In one embodiment, a first data processing thread pool includes at least one data processing thread, which is used to execute data processing tasks assigned to the first data node; the processing unit determines the data processing tasks executed in the first data processing thread pool as the first data processing task to be inspected in the distributed database system, including:

[0019] Retrieve the data processing record corresponding to each data processing thread in the first data processing thread pool. The data processing record includes the data processing statement executed by the corresponding data processing thread.

[0020] The data processing tasks represented by the data processing statements executed by each data processing thread in the first data processing thread pool are identified as the first data processing tasks to be inspected in the distributed database system.

[0021] In one embodiment, the first data processing thread pool contains at least one data processing thread, and each data processing thread in the first data processing thread pool corresponds to its own data processing record; the acquisition unit acquires the execution information of N first data processing tasks to be inspected in the distributed database system, including:

[0022] Set the data processing records corresponding to each data processing thread in the first data processing thread pool as execution information.

[0023] In one embodiment, the execution information includes a data processing record corresponding to each data processing thread in the first data processing thread pool, and the data processing record includes the data processing statements executed by the corresponding data processing thread; any one of the N first data processing tasks is represented as the first data processing task N. i , where i is a positive integer less than or equal to N;

[0024] The processing unit analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0025] Based on the data processing statements contained in the execution information, calculate the first data processing task N. i The number of data processing threads used in the first data processing thread pool;

[0026] Set the quantity to N for the first data processing task. i Resource consumption data during execution;

[0027] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the number of data exceeding the preset thread consumption threshold.

[0028] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. The data processing records include: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread; any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N;

[0029] The processing unit analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0030] Based on the data processing statements contained in the execution information, determine the first data processing task N. i The first data processing thread occupied in the first data processing thread pool;

[0031] Obtain the resource utilization rate of the first data processing thread in the first data node from the stack snapshot information of the first data processing thread;

[0032] Set the resource utilization rate to the first data processing task N. i Resource consumption data during execution;

[0033] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: resource utilization rate exceeding the preset resource utilization rate threshold.

[0034] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. The data processing records include: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread. The stack snapshot information includes the method call chain. Any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N;

[0035] The processing unit analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0036] Based on the data processing statements contained in the execution information, determine the first data processing task N. i The first data processing thread occupied in the first data processing thread pool;

[0037] The stack snapshot information of the first data processing thread includes the number of calls to various methods in the method call chain;

[0038] Set the maximum number of calls among the statistically obtained call counts as the first data processing task N. i Resource consumption data during execution;

[0039] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the maximum number of calls exceeding the preset call count threshold.

[0040] In one embodiment, the processing unit intervenes in the execution process of the second data processing task, including:

[0041] If the second data processing task meets the task cancellation conditions, then the execution process of the second data processing task will be stopped.

[0042] The conditions for canceling the second data processing task include any of the following: the execution progress of the second data processing task has not reached the preset execution progress, the priority of the second data processing task is lower than the preset priority, the second data processing task has no associated business process, or the distributed database system has the task cancellation switch turned on.

[0043] In one embodiment, the second data processing task is divided into at least one subtask, and one subtask is executed by a data processing thread in the distributed database system; the processing unit stops the execution of the second data processing task, including:

[0044] Send corresponding cancellation instructions to the data processing threads that are executing at least one subtask, respectively, to indicate that the corresponding subtasks should be cancelled.

[0045] In one embodiment, the processing unit intervenes in the execution process of the second data processing task, including:

[0046] If the second data processing task meets the task interception conditions, then instruct the data nodes in the distributed database system to intercept the second data processing task.

[0047] The conditions for the second data processing task to meet the task interception criteria include: the second data processing task has no associated business process, the priority of the second data processing task is lower than the preset priority, and the task interception switch of the distributed database system is turned on.

[0048] In one embodiment, the processing unit intervenes in the execution process of the second data processing task, including:

[0049] Send the execution log information of the second data processing task to the management terminal of the distributed database system. The execution log information is used to indicate one or more of the following: the processing logic adjustment strategy of the second data processing task, the data distribution adjustment strategy of the distributed database system, and the business process adjustment strategy of the second data processing task.

[0050] In one embodiment, the method for detecting the arrival of the inspection time includes at least one of the following:

[0051] When the preset inspection cycle is reached, the inspection time is determined.

[0052] When the frequency of task requests in the distributed database system exceeds a preset frequency threshold, the inspection time is determined.

[0053] When the resource utilization rate of a distributed database system exceeds a preset utilization rate threshold, the inspection time is determined.

[0054] When the task execution failure rate of a distributed database system exceeds a preset failure rate threshold, the inspection time is determined.

[0055] On one hand, embodiments of this application provide a computer device, the computer device comprising:

[0056] Processor, memory, and network interface;

[0057] The processor is connected to the memory and the network interface. The network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to implement the above-mentioned task processing method.

[0058] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including computer instructions adapted to be loaded by a processor and executed by the above-described task processing method.

[0059] On the one hand, embodiments of this application provide a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the above-mentioned task processing method is implemented.

[0060] The task processing scheme provided in this application embodiment can obtain the execution information of N first data processing tasks to be inspected in the distributed database system when the inspection time arrives. These N first data processing tasks are determined based on the inspection mechanism of the distributed database system. This mechanism allows for dynamic planning of the inspection scope, significantly improving the task inspection efficiency of the distributed database system. Statistical analysis of the execution information yields resource consumption data for each first data processing task. Based on this data, second data processing tasks are identified from the N first data processing tasks, where the resource consumption data exceeds a preset threshold. Through statistical analysis of the execution information, high-resource-consuming second data processing tasks in the distributed database system can be identified, thus pinpointing high-load data processing tasks that may cause load spikes and compromise customer SLAs. Intervention processing is applied to the execution process of these second data processing tasks. This intervention can promptly eliminate potential load risks associated with these tasks, improving the manageability of the distributed database system for these tasks and ultimately enhancing the stability and availability of the distributed database system. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0062] Figure 1 This is a schematic diagram of a task interception method provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of a task processing scenario provided in an embodiment of this application;

[0064] Figure 3 This is a flowchart illustrating a task processing scheme provided in an embodiment of this application;

[0065] Figure 4 This is a flowchart illustrating a task processing method provided in an embodiment of this application;

[0066] Figure 5 This is a flowchart illustrating a task localization method provided in an embodiment of this application. Figure 1 ;

[0067] Figure 6 This is a flowchart illustrating a task localization method provided in an embodiment of this application. Figure 2 ;

[0068] Figure 7 This is a schematic diagram of the structure of a task processing device provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0071] To better understand the task processing scheme provided in the embodiments of this application, the key terms involved in this scheme will be introduced first:

[0072] I. Distributed Database Systems:

[0073] A Distributed Database System (DDBS) is a system that distributes data storage and management across multiple nodes. These nodes work together to manage and access data. In other words, a DDBS can leverage the collaborative work of multiple nodes to execute data processing tasks in parallel. Elasticsearch (ES, a distributed search and analytics engine) is a typical example of a DDBS. Its distributed architecture, built on Lucene (an open-source full-text search engine toolkit), is naturally suited for scenarios involving the storage, retrieval, and analysis of massive amounts of data, such as log analysis, full-text search, structured search, complex queries, and data visualization.

[0074] For example, distributed database systems typically possess the following characteristics: 1) Distributed data storage: Data can be distributed across different nodes according to a certain partitioning strategy, with each node storing a portion of the data (called a shard). This allows multiple nodes to operate on their respective shards simultaneously during data processing, achieving parallel processing. 2) Parallel processing capability: When a data processing task is received, it can be split and distributed across multiple nodes for simultaneous execution. For instance, a complex query task involving multiple shards can be broken down into multiple subtasks, each corresponding to a shard. These subtasks are sent to the nodes storing their respective shards, where they are processed in parallel before the results are aggregated and returned. 3) Coordination mechanism: Nodes can coordinate their operations through special software protocols and communication mechanisms.

[0075] II. Data Processing Tasks:

[0076] A data processing task refers to a set of data operations collaboratively completed by various nodes in a distributed database system. A set of data operations can consist of at least one data operation (usually two or more), and one data operation within the set can be viewed as a sub-task formed by breaking down the data processing task. For example, the types of data operations covered by this set may include, but are not limited to: data storage operations, data deletion operations, data query operations, and data computation operations.

[0077] Data storage operations can include one or more of the following: data sharding, data replication, data update, and data insertion. Data sharding involves dividing large-scale data (data exceeding a preset limit) into multiple shards according to certain rules (such as hash sharding or range sharding), and storing each shard on different nodes. Data replication involves copying data to multiple nodes to ensure that if one node fails, other nodes can take over its responsibilities. Data update involves modifying or overwriting existing data, such as updating a field in a record. Data insertion involves adding new data to a distributed database system. Data deletion involves removing specified data from a distributed database system. Data query operations can include one or more of the following: simple queries and complex queries. Simple queries perform fast searches based on data on a single node. Complex queries involve data on multiple nodes and require cross-node joins, such as multi-table joins and aggregation queries. Data computation operations can involve statistical calculations on large-scale data, such as summation, averaging, maximum, and minimum value calculations, and can also involve online analysis and processing of real-time streaming data.

[0078] III. Nodes:

[0079] A node is the basic working unit that constitutes a distributed database system. It can be an independent server, virtual machine, or container instance, and this application does not impose any restrictions on it. Each node can assume different roles, and the types of nodes may include, but are not limited to, master nodes, data nodes, and coordinating nodes.

[0080] The master node is responsible for managing the global state of a distributed database system. Tasks such as creating or deleting indexes and determining which node a shard is assigned to all rely on the master node for coordination and decision-making. Furthermore, the master node is typically responsible for the initial configuration when a new node joins and for coordinating the departure of a node. For large distributed database systems, to ensure stable operation, the master node usually does not participate in routine data processing tasks, focusing instead on system management and coordination.

[0081] In distributed database systems, indexes are typically a way to logically group documents, with each index considered an independent logical space. For example, in an e-commerce distributed database system, one index can be created to store all product-related documents, and another to store all customer-related documents. This logical division allows for the categorization and management of different types of data. An index can be exemplified by two parts: an index structure and index data, with distributed physical storage management achieved through sharding. The index structure defines the document storage rules and retrieval logic. For instance, it can store key information about the document (such as the document ID, field values, etc.) and the mapping between this key information and the document's storage location, enabling the distributed database system to efficiently locate and retrieve the required document. Index data refers to the actual stored document content and its metadata (such as the document ID).

[0082] Sharding refers to dividing index data into multiple independent, movable parts, allowing the index data to be distributed across multiple data nodes for storage and processing. In distributed database systems, there are various sharding strategies; for example, a document's ID (identity) can be hashed, and the result can be modulo the number of shards to determine which shard a document should be assigned to. Another example is in an e-commerce distributed database system, where product data for electronics can be stored on one shard, and product data for clothing on another. Shards can be categorized into primary shards and replica shards. A replica shard is a copy of the primary shard, used to provide data redundancy when the primary shard fails. When the node hosting a primary shard fails, a replica shard can automatically be promoted to primary, continuing to provide data services and ensuring the availability and security of the distributed database system.

[0083] Data nodes are the carriers for storing shards and can perform shard-related operations, such as writing, querying, and computing shards. Coordinating nodes are responsible for receiving and processing data processing tasks submitted to the distributed database system by clients. For example, the submitter of a data processing task can write data processing statements that conform to the requirements of the distributed database system, according to their own data processing needs. In Elasticsearch, for instance, the submitter can use a DSL (Domain Specific Language) to write data processing statements. A data processing statement is a set of instructions written to implement a data processing task, describing which data operations need to be performed and how to perform them; that is, the data processing task is represented in the form of data processing statements. When a data processing statement is submitted to the coordinating node, the coordinating node is responsible for parsing the statement, determining the shard locations involved in the data processing task, and identifying at least one subtask into which the data processing task is divided. Based on the shard locations, the coordinating node routes the subtasks to the corresponding data nodes. These data nodes can return the results of the corresponding subtasks to the coordinating node, which then merges and organizes the results before returning them to the submitter of the data processing task.

[0084] IV. Data Processing Thread Pool:

[0085] A thread pool is a multithreading mechanism that manages a pre-created number of threads by placing them in a pool. When a task needs to be executed, instead of creating a new thread each time, an idle thread is taken from the thread pool to execute the task. After the task is completed, the thread returns to the thread pool to wait for the next task assignment. A thread pool typically includes a task queue, which stores tasks submitted to the thread pool but not yet executed.

[0086] A data processing thread pool is a thread pool mechanism in a distributed database system used to execute data processing tasks. One or more data processing threads can be pre-created in the data processing thread pool. The types of data processing thread pools can vary. For example, the types include, but are not limited to: 1) Generic Thread Pool: used for handling general operations, such as node discovery and heartbeat detection. 2) Search Thread Pool: used for handling query-related operations, such as counting the number of documents that meet certain conditions and retrieving data from the index based on those conditions. 3) Index Thread Pool: used for handling indexing operations (inserting documents into the index) and deletion operations (removing documents from the index). 4) Suggest Thread Pool: used for performing suggestion operations (providing supplementary suggestions based on input prefixes). 5) Bulk Thread Pool: used for performing batch data operations, such as inserting multiple documents at once, updating multiple documents at once, and deleting multiple documents at once.

[0087] In the operation and maintenance of distributed database systems, it often happens that the total number of tasks (the total number of all data processing tasks that the distributed database system needs to process per unit time) remains almost constant, but the proportion of large tasks (such as complex query tasks, large-scale data import or export tasks, long-running data processing tasks, etc.) increases, leading to a sudden surge in load on the distributed database system. One solution is to address this by exhausting the resources of the data processing thread pool—that is, when there are no idle data processing threads in the pool and the task queue is full—tasks already submitted to the data processing thread pool can be queued in the task queue to wait for processing, while subsequent submissions will be directly rejected. For example, please refer to [link to relevant documentation]. Figure 1This is a schematic diagram of a task interception method provided in an embodiment of this application. When the task queue of the search thread pool is full, subsequent query tasks sent to the search thread pool will directly throw an exception. In another solution, the QPS (Queries Per Second, the number of tasks received per unit time) of the distributed database system can be obtained. When the QPS exceeds a preset QPS threshold (which can be set as needed), the processing of some or all subsequent tasks can be intercepted. This solution, on the one hand, rejects a large number of normal data processing tasks, leading to serious damage to the customer's SLA; on the other hand, it cannot identify large tasks, impairing the stability and availability of the distributed database system. In yet another solution, after receiving a data processing task, the execution cost, complexity, and efficiency of the data processing task can be determined by analyzing the execution plan, the size of the documents involved, and the index situation, thereby intercepting large tasks in advance. This solution has low versatility; for example, for Elasticsearch, it is impossible to evaluate CPU overhead solely based on the DSL statement and the size of the indexes involved in the DSL statement. While developers can use flame graphs (a visual performance analysis tool that can intuitively display information such as resource consumption (CPU, memory) and method call chains during the operation of a distributed database system) to see problems such as high operating load pressure in the distributed database system, they cannot pinpoint the exact cause of the problem. They need to search through a large number of logs to find out which type of large task caused the problem. As a result, distributed database systems suffer from time-consuming problem analysis and difficulty in resolving problems in a timely manner.

[0088] This application proposes a task processing scheme that can dynamically plan the inspection scope through an inspection mechanism, significantly improving the task inspection efficiency of a distributed database system. In addition, by statistically analyzing the execution information of the data processing tasks to be inspected, high-load data processing tasks that may cause load surges and damage to customer SLAs in the distributed database system can be automatically identified and intervened on. Through intervention, potential load risks in the distributed database system can be eliminated in a timely manner, improving the manageability of the distributed database system for data processing tasks, thereby achieving the goal of enhancing the stability and availability of the distributed database system.

[0089] The following describes the task processing scenarios to which the task processing scheme provided in the embodiments of this application is applicable. Please refer to [link to relevant documentation]. Figure 2This is a schematic diagram of a task processing scenario provided in an embodiment of this application. The task processing scenario includes a client 21 (the number of clients 21 can be one or more), a distributed database system 22, and a system inspection node 23. A communication connection is established between the client 21 and the distributed database system 22. The system inspection node 23 can be the master node in the distributed database system 22, or any node in the distributed database system 22, or something else entirely. Figure 2 The computer devices shown have established communication connections with each node in the distributed database system 22 (i.e., Figure 2 System inspection node 23 (in the document) is a computer device. The computer device can be a terminal device, such as a smartphone, tablet, smart wearable device, smart voice interaction device, vehicle terminal, smart home appliance, personal computer, etc. The computer device can also be a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The communication connection can be wired or wireless; this application does not impose any restrictions.

[0090] Client 21 is the initiator of the data processing task. For example, client 21 can provide a visual interface that can receive input data processing statements. Client 21 can send the input data processing statements to distributed database system 22, and distributed database system 22 can execute the data processing task according to the data processing statements. After the data processing task is completed, distributed database system 22 can return the task processing result to client 21.

[0091] System inspection node 23 can be used to execute the task processing scheme provided in this application. This task processing scheme is applied to the distributed database system 22. Please refer to [link / reference]. Figure 3 This is a flowchart illustrating a task processing scheme provided in an embodiment of this application. The general flow of the task processing scheme includes the following steps S31-S34:

[0092] S31. When the inspection time of the distributed database system is reached, obtain the execution information of the N first data processing tasks to be inspected in the distributed database system, where N is a positive integer.

[0093] The inspection time of a distributed database system refers to the point in time at which the operational load of the distributed database system is monitored. The operational load of a distributed database system reflects the workload it experiences at a given moment. For example, the operational load may include at least one of the following: CPU load (e.g., CPU utilization), memory load (e.g., memory utilization), and disk I / O (Input / Output) load (e.g., disk I / O read / write rate). CPU utilization refers to the proportion of time the CPU is used to execute tasks within a certain period; high CPU utilization indicates CPU resource scarcity, potentially leading to a large number of tasks queuing for processing and slowing down the execution speed of new tasks. Memory utilization refers to the proportion of currently used physical memory to total available memory; excessively high memory utilization may result in frequent data exchange between memory and disk, potentially slowing down task processing. Disk I / O read / write rate describes the amount of data that the disk can successfully transfer per unit of time; high read / write rates mean the disk is frequently transferring data, which may lead to disk I / O bottlenecks and affect the execution of tasks that rely on disk data.

[0094] In distributed database systems, a large number of data processing tasks are typically running concurrently. To improve task inspection efficiency, N primary data processing tasks to be inspected can be identified from the distributed database system. Since large tasks usually involve complex calculations and logical processing, as well as frequent disk I / O operations, they are likely to increase the operating load of the distributed database system. Therefore, these N primary data processing tasks can be data processing tasks executed by high-load nodes in the distributed database system. High-load nodes are those whose resource consumption is excessive, causing their performance to be significantly affected. By narrowing the scope of data processing task inspection in this way, the task inspection efficiency of the distributed database system can be significantly improved.

[0095] The execution information of N first data processing tasks refers to various information related to the execution process of these N first data processing tasks, such as which data processing threads are used to execute the first data processing tasks, and the stack snapshot information of these data processing threads when executing the first data processing tasks. Stack snapshot information refers to a "photograph" record of the call stack state at a certain moment (called the snapshot moment). The call stack is a data structure that stores method call information during thread execution. For example, stack snapshot information can include the following aspects: 1) The structure and hierarchy of the call stack, which can present the method call chain of the data processing thread up to the snapshot moment. The method call chain can be used to describe the currently executing method (i.e., the method being executed at the snapshot moment), and which methods have been called in sequence; the methods in the method call chain are a code block that encapsulates a specific function. 2) Information related to each method, such as the parameters and values ​​of local variables, reflects the specific data situation during method execution; 3) The status (such as congestion state, running state, waiting state, etc.) and information (such as the ID of the data processing thread) of the currently executing task can be used to determine whether the data processing thread is working normally. For example, a data processing thread that is in a waiting state for a long time may have a resource deadlock problem. Resource deadlock refers to two or more processes (or threads) waiting for each other to release the resources they hold, causing these processes (or threads) to be unable to continue, thus falling into a stalemate. 4) Debugging-related information, such as the current location of the executing code. 5) Performance metrics, such as method execution time, CPU utilization, memory utilization, and disk I / O data (such as disk read volume, disk write volume, and number of disk I / O operations).

[0096] S32. Analyze the execution information to obtain the resource consumption data when each first data processing task is executed.

[0097] Resource consumption data during the execution of the first data processing task refers to a quantitative description of the usage of various resources during the execution of the first data processing task. For example, resource consumption data may include the number of data processing threads occupied by the first data processing task, the CPU utilization and memory utilization of the data processing threads executing the first data processing task, and the disk I / O data caused by the data processing threads executing the first data processing task.

[0098] S33. Based on the resource consumption data when each first data processing task is executed, locate the second data processing task from the N first data processing tasks. The amount of resource consumption data when the second data processing task is executed is higher than the preset consumption threshold.

[0099] The preset consumption threshold is used to measure whether the amount of resource-consuming data is within a reasonable range. It can be determined based on historical resource consumption data or experience. For example, it can be set based on experience: if the number of data processing threads used by the first data processing task exceeds 100, or the CPU utilization of the first data processing task exceeds 70%, or the memory utilization of the first data processing task exceeds 80%, then the first data processing task is designated as the second data processing task. When the amount of resource-consuming data during the execution of the second data processing task exceeds the preset consumption threshold, this second data processing task is likely a large task and may be the cause of a sudden increase in the load on the distributed database system. Therefore, intervention can be performed on the execution process of the second data processing task.

[0100] S34. Intervene in the execution process of the second data processing task.

[0101] For example, the intervention method may include at least one of the following: canceling the execution of the second data processing task; adding the data processing statement corresponding to the second data processing task to the task interception list of the distributed database system; sending the execution log information of the second data processing task to the management terminal, so that developers can quickly and accurately locate the problems that occur during the execution of the second data processing task through the execution log information in the management terminal, such as determining whether there are logical errors, resource competition or deadlock.

[0102] Therefore, this application selects data processing tasks executed by high-load nodes in the distributed database system as inspection targets. This allows us to focus on data processing tasks that may have a significant impact on the performance of the distributed database system, avoiding ineffective inspections of a large number of low-load tasks and greatly improving the efficiency of task inspection. At the same time, by analyzing resource consumption data and intervening in problematic tasks, the distributed database system can more rationally adjust the allocation of system resources, improve the utilization efficiency of system resources, and further enhance the stability and availability of the distributed database system.

[0103] In this application embodiment, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations. Obtaining personal information requires the knowledge or consent of the individual (or a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws, regulations, and the authorization of the personal information subject. For example, when this application embodiment is applied to specific products or technologies, such as obtaining data processing statements, permission or consent from the submitter of the data processing statements is required, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0104] Based on the task processing scheme described above, this application proposes a more detailed task processing method. Please refer to [link to relevant documentation]. Figure 4 This is a flowchart illustrating a task processing method provided in an embodiment of this application; the task processing method can be executed by the system inspection node 23, and the task processing method is applied to a distributed database system (such as the distributed database system 22 described above). The task processing method may include, but is not limited to, steps S41-S44:

[0105] S41. When the inspection time of the distributed database system arrives, determine the N first data processing tasks to be inspected from the distributed database system. The first data processing task is the data processing task executed in the first data processing thread pool, and N is a positive integer.

[0106] In one implementation, the method for detecting the arrival of inspection time includes at least one of the following: 1) Determining the arrival of inspection time when a preset inspection cycle is reached. The inspection cycle is a fixed time interval. For example, if the inspection cycle is preset to 10 minutes, then the inspection cycle can indicate that the inspection time of the distributed database system arrives every 10 minutes. 2) Determining the arrival of inspection time when the task request frequency of the distributed database system is higher than a preset frequency threshold. The task request frequency refers to the number of data processing tasks received by the distributed database system per unit time, reflecting the current busy level of the distributed database system. The preset frequency threshold can be a value determined based on factors such as the task design capacity of the distributed database system, the historical task load, and business needs. For example, in the past year of operation, the distributed database system of an e-commerce platform received an average of 8,000-10,000 data processing tasks per hour during peak business hours. Therefore, the preset frequency threshold can be set to 8,000 task requests per hour. When the task request frequency of the distributed database system is higher than the preset frequency threshold, it indicates that the load pressure of the distributed database system is increasing, and it may face performance pressure or potential problems. At this time, inspection is required to ensure the stable operation of the distributed database system. 3) When the resource utilization rate of the distributed database system exceeds a preset utilization threshold, the inspection time is determined. This resource utilization rate can be any of the following: memory utilization, CPU utilization, or disk I / O utilization (reflecting the proportion of time the disk spends processing I / O tasks within a given period). The preset utilization threshold can be a reasonable usage limit determined based on system hardware configuration, performance requirements, etc. When the resource utilization rate of the distributed database system exceeds the preset utilization threshold, it indicates that system resources are under strain, which may affect the normal execution of tasks. Inspection at this time helps to promptly identify resource bottlenecks and take corresponding measures. 4) When the task execution failure rate of the distributed database system exceeds a preset failure rate threshold, the inspection time is determined. The task execution failure rate refers to the ratio of the number of failed tasks to the total number of tasks in the distributed database system within a certain period. The preset failure rate threshold can be an acceptable failure rate range determined based on the reliability requirements of the distributed database system. When the task execution failure rate of the distributed database system exceeds the preset failure rate threshold, it indicates that the distributed database system may have some problems, such as unbalanced data distribution or insufficient resources, requiring a comprehensive inspection to find the root cause and resolve the problem.

[0107] A distributed database system may include multiple data nodes, each running a data processing thread pool. For example, each data node's data processing thread pool may include, but is not limited to, a general thread pool, a search thread pool, an index thread pool, a suggestion thread pool, and a batch thread pool. A data processing task in a distributed database system can be assigned to a data processing thread pool on at least one data node for execution. For instance, at least one shard involved in the data processing task can be identified, and the task can be broken down into at least one subtask. Each subtask typically corresponds to one shard. These subtasks can be assigned to data nodes containing the corresponding shards. Data nodes can then assign subtasks to appropriate data processing thread pools based on their type; for example, query-related subtasks are assigned to the search thread pool, and index-related subtasks are assigned to the index thread pool. Since data nodes are the specific executors of data processing tasks, their performance directly affects the overall performance of the distributed database system. For example, if the CPU utilization (or memory utilization, disk I / O read / write speed) of a data node is too high, the processing speed will be significantly slower when handling new data processing tasks. Since the response time of a distributed database system is determined by the processing status of each data node, a slowdown in the processing speed of one or more data nodes will lead to an extension of the overall response time of the distributed database system, thereby affecting the overall performance of the distributed database system.

[0108] In one embodiment, determining N first data processing tasks to be inspected from a distributed database system includes the following steps 1-3:

[0109] Step 1: Obtain the operating load of each data node. The operating load may include at least one of the following: CPU load, memory load, and disk I / O load.

[0110] Step 2: Based on the operating load of each data node, determine a first data node from multiple data nodes. The operating load of the first data node is higher than a preset load threshold. The first data node runs a first data processing thread pool. The number of first data nodes can be one or more. The preset load threshold is used to measure whether the operating load of the data node is within a reasonable range. For example, when the operating load of the data node includes CPU load, the preset load threshold includes a preset CPU load threshold, which is used to measure whether the CPU load of the data node is within a reasonable range; when the operating load of the data node includes memory load, the preset load threshold includes a preset memory load threshold, which is used to measure whether the memory load of the data node is within a reasonable range; when the operating load of the data node includes disk I / O load, the preset load threshold includes a preset disk I / O load threshold, which is used to measure whether the disk I / O load of the data node is within a reasonable range. The preset load threshold can be determined based on experience or historical operating load. For example, for a distributed database system, if the operating load of each data node is continuously collected over a period of time, including CPU utilization, memory utilization, and I / O read / write frequency, analysis may reveal that during periods of stable and high-performance operation of the distributed database system, the average CPU utilization of most data nodes is around 50%, the average memory utilization is around 60%, and the I / O read / write frequency is between 100 and 200 times per second. Based on experience, the preset CPU load threshold can be set to 70% CPU utilization, the preset memory load threshold can be set to 75% memory utilization, and the preset disk I / O load threshold can be set to 300 I / O read / write frequency per second.

[0111] Step 3: Identify the data processing tasks executed in the first data processing thread pool as the first data processing tasks to be inspected in the distributed database system. In other words, the first data processing thread pool is a data processing thread pool running on high-load data nodes, and the data processing tasks executed in the first data processing thread pool can be identified as the first data processing tasks to be inspected in the distributed database system.

[0112] In one embodiment, multiple data processing thread pools (such as a general thread pool, search thread pool, index thread pool, suggestion thread pool, and batch thread pool) may run on the first data node. In one implementation, all of the multiple data processing thread pools running on the first data node can be designated as the first data processing thread pool. In another implementation, a pre-defined task inspection target can be obtained. The task inspection target can indicate the type of large task to be inspected. The data processing thread pool related to the task inspection target can be determined from the multiple data processing thread pools running on the first data node and designated as the first data processing thread pool. For example, if the task inspection target indicates that the type of large task to be inspected is a large query task, and since the search thread pool has the capability for large-scale data querying, it is usually used to execute sub-tasks formed after the large query task is broken down, the search thread pool running on the first data node can be designated as the data processing thread pool related to the task inspection target (i.e., the first data processing thread pool). A large query task refers to a query task that significantly exceeds the scope of conventional operations in terms of the scale of the query data and the complexity of the query operation; a query task refers to a data processing task involving data query operations. Similarly, if the task inspection target indicates that the large task to be inspected is a large storage task, then the index thread pool and batch thread pool running on the first data node can be used as the data processing thread pool related to the task inspection target (i.e., as the first data processing thread pool). Large storage tasks refer to those storage tasks that significantly exceed the scope of regular operations in terms of data storage scale and storage operation complexity; storage tasks refer to data processing tasks involving data storage operations. In another implementation, since large tasks often involve frequent disk I / O operations, as well as complex calculations and logical processing, the data processing thread pools executing CPU-intensive tasks (such as search thread pools and index thread pools) and those executing I / O-intensive tasks (such as batch thread pools) on the first data node can be identified as the first data processing thread pool. This allows focus on the key data processing thread pools for large task execution. CPU-intensive tasks refer to tasks that mainly rely on the computing power of the Central Processing Unit (CPU) and typically involve frequent computational operations. I / O-intensive tasks refer to tasks primarily constrained by input / output operations (e.g., writing documents to disk, reading data from disk, etc.) rather than computational power. In another implementation, the runtime load (e.g., CPU load, memory load, disk I / O load) of each data processing thread pool running on the first data node can be obtained. Data processing thread pools whose loads on the first data node exceed a reference load threshold are identified as the primary data processing thread pool, allowing for focused attention on high-load data processing thread pools.The reference load threshold can be determined based on experience or historical operating load. For example, collect the operating load of the data processing thread pool on the first data node over a period of time. If the analysis shows that the average load of the data processing thread pool is 60%, then the reference load threshold can be set to 70%-80%.

[0113] In one implementation, a first data processing thread pool contains at least one data processing thread. These threads execute data processing tasks assigned to the first data node. Determining the data processing tasks executed in the first data processing thread pool as the first data processing task to be inspected in the distributed database system includes: obtaining the data processing record corresponding to each data processing thread in the first data processing thread pool. The data processing record includes the data processing statements executed by the corresponding data processing thread. For example, ThreadLocal (thread-local variables) is a utility class provided by Java (a high-level programming language) that provides each thread using ThreadLocal with an independent copy of the variable. This means that each thread can independently modify its own copy of the variable without affecting the copies of other threads. The data processing record corresponding to each data processing thread can refer to the information recorded by each data processing thread in its own ThreadLocal. The data processing threads in the first data processing thread pool can obtain the data processing statements of the assigned data processing tasks and store these statements in their own ThreadLocal. For example, after the client sends a data processing statement (such as a DSL statement) to the coordinating node, the coordinating node can use the data processing statement to break down the data processing task represented by the statement into at least one subtask. These subtasks can be distributed to the corresponding data nodes along with the data processing statement. When the data processing threads on these data nodes execute the subtasks, they can store the data processing statement carried by the subtasks in their own ThreadLocal. The data processing task represented by the data processing statement executed by each data processing thread in the first data processing thread pool can be identified as the first data processing task to be inspected in the distributed database system. In other words, the first data processing task refers to the data processing task executed by each data processing thread in the first data processing thread pool.

[0114] S42. Obtain the execution information of N first data processing tasks. The execution information is determined based on the data processing records corresponding to each data processing thread in the first data processing thread pool.

[0115] In one embodiment, a first data processing thread pool contains at least one data processing thread, and each data processing thread in the first data processing thread pool corresponds to its own data processing record. The data processing records corresponding to each data processing thread in the first data processing thread pool can be set as execution information for N first data processing tasks. For example, the data processing record may include the data processing statements executed by the corresponding data processing thread, thus revealing the data processing task executed by each data processing thread. For example, the data processing record may include the stack snapshot information of the corresponding data processing thread, thus revealing some information about each data processing thread when executing its corresponding subtask. In a feasible implementation, the stack snapshot information of the data processing thread may be generated and stored when the inspection time of the distributed database system arrives, or it may be generated when the data processing thread encounters an unhandled exception or crash, or it may be generated periodically by the data processing thread (e.g., every 5 minutes). Therefore, this application focuses on the execution status of these first data processing tasks on high-load data nodes.

[0116] S43. Analyze the execution information to obtain the resource consumption data when each first data processing task is executed. Based on the resource consumption data when each first data processing task is executed, locate the second data processing task from the N first data processing tasks. The amount of resource consumption data when the second data processing task is executed is higher than the preset consumption threshold.

[0117] Any one of the N first data processing tasks is represented as the first data processing task N. i Where i is a positive integer less than or equal to N, the following uses the first data processing task N as an example. i Let's take S43 as an example to explain.

[0118] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool, and the data processing records include the data processing statements executed by the corresponding data processing thread; the execution information is analyzed to obtain the first data processing task N. i Resource consumption data during execution includes: statistics on the first data processing task N based on the data processing statements contained in the execution information. i The number of data processing threads used in the first data processing thread pool. For example, there are two first data processing thread pools, which together include: data processing thread A, data processing thread B, data processing thread C, data processing thread D, and data processing thread E. The data processing statements executed by data processing threads A, B, and C belong to the first data processing task N. i The first data processing task Ni The first data processing thread pool contains 3 data processing threads. The first data processing task N can be... i The number of data processing threads used in the first data processing thread pool is set to N for the first data processing task. i Resource consumption data during execution. Correspondingly, the first data processing task N... i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the first data processing task N. i The number of data processing threads in the first data processing thread pool exceeds the preset thread consumption threshold. In other words, when the first data processing task N... i When the number of data processing threads in the first data processing thread pool exceeds a preset thread consumption threshold, the first data processing task N can be... i The task is designated as the second data processing task. The preset thread consumption threshold is the upper limit of the number of data processing threads normally occupied by a data processing task in the first data processing thread pool. For example, it can be set based on the data processing thread usage of data processing tasks over a period of time. For instance, if the number of data processing threads occupied by similar data processing tasks in the first data processing thread pool over a period of time (such as a week or a month) is 10-15, then 16 can be used as the preset thread consumption threshold.

[0119] For example, please see Figure 5 This is a flowchart illustrating a task localization method provided in an embodiment of this application. Figure 1 After the DSL statement of the query task sent by the client arrives at the distributed database system, it will perform query operations on shard 1 and shard 2 in data node 1 according to the query link, and perform data query operations on shard 3 and shard 4 in data node 2. The data query operations can be performed by search threads. When performing data query operations, the search thread will store the DSL statement in its own ThreadLocal. Therefore, the number of search threads occupied by the query task can be counted by the DSL statements recorded by each search thread. When the number of search threads occupied by the query task is higher than the preset thread consumption threshold, the query task can be identified as the second data processing task.

[0120] In one feasible implementation, the first data processing task that occupies the largest number of data processing threads in the first data processing thread pool can be determined from N first data processing tasks and designated as the second data processing task. Therefore, this embodiment of the application can identify those first data processing tasks that consume a large number of data processing threads in the first data processing thread pool and designate them as the second data processing task.

[0121] Large tasks, due to their high complexity, long execution time, and high resource consumption, are prone to generating hot stacks. Therefore, the data processing thread executing the hot stack can be identified as the hot thread, and the data processing task executed by the hot thread can be identified as the large task (i.e., the second data processing task). A hot stack refers to the frequently accessed or time-consuming portion of the call stack. For example, a hot stack has at least one of the following characteristics: frequent method calls; long execution time, possibly due to complex computational logic, extensive data processing, or time-consuming interactions with external resources (such as databases, file systems, networks, etc.); and high-frequency calls or long execution times consume significant system resources, especially CPU and memory, thus often causing performance limitations or degradation in distributed database systems.

[0122] For example, please see Figure 6 This is a flowchart illustrating a task localization method provided in an embodiment of this application. Figure 2 After the DSL statement of the query task sent by the client arrives at the distributed database system, it will perform query operations on shard 1 and shard 2 in data node 1 according to the query chain, and perform data query operations on shard 3 and shard 4 in data node 2. The data query operation can be executed by the search thread. When the search thread performs the data query operation, it can store the stack snapshot information of the search thread (used to record the call stack state) in the ThreadLocal of the search thread. In this way, it is possible to detect whether the search thread is a hot thread based on the stack snapshot information of the search thread. When the search thread is detected to be a hot thread, the query task can be identified as the second data processing task.

[0123] The following uses the first data processing task N as an example. i For example, several task localization methods based on hotspot stack detection are demonstrated. The execution information includes the data processing record for each data processing thread in the first data processing thread pool. This record includes the data processing statements executed by the corresponding data processing thread and the corresponding stack snapshot information.

[0124] In one implementation, the execution information is analyzed to obtain the first data processing task N. i Resource consumption data during execution includes: determining the first data processing task N based on the data processing statements contained in the execution information. i The first data processing thread in the first data processing thread pool, that is, the data processing statement executed by the first data processing thread to represent the first data processing task N. iThe resource utilization (such as memory utilization or CPU utilization) of the first data processing thread in the first data node can be obtained from the stack snapshot information of the first data processing thread. The resource utilization of the first data processing thread in the first data node can be set as the resource utilization of the first data processing task N. i Resource consumption data during execution. Among them, the first data processing task N... i The data volume of resource consumption data during execution exceeding the preset consumption threshold includes: the resource utilization rate of the first data processing thread in the first data node exceeding the preset resource utilization rate threshold. In other words, when the resource utilization rate of the first data processing thread in the first data node exceeds the preset resource utilization rate threshold, the first data processing task N can be... i This task is designated as the second data processing task. The preset resource utilization threshold is the upper limit of the resource utilization rate normally consumed by a data processing task in the first data node. For example, the preset resource utilization threshold can be set based on the resource utilization rate of the data processing task over a period of time. For instance, if the average resource utilization rate of the data processing task over the past week is 40%, 5% can be set as the preset resource utilization threshold.

[0125] In one feasible implementation, the first data processing task with the highest resource utilization in the first data node can be determined from N first data processing tasks and used as the second data processing task.

[0126] In another implementation, the execution information is analyzed to obtain the first data processing task N. i Resource consumption data during execution includes: determining the first data processing task N based on the data processing statements contained in the execution information. i The first data processing thread is used in the first data processing thread pool. The execution time of the first data processing thread can be determined based on its stack snapshot information. For example, the stack snapshot information of the first data processing thread can include the method call chain. The start and end times of each method in the method call chain can be recorded in the relevant information of each method. When the currently executing method in the method call chain has not yet finished executing, the duration of the time interval from the start time of the currently executing method to the current time point can be determined as the execution time of the first data processing thread. When the currently executing method in the method call chain has finished executing, the duration of the time interval from the start time to the end time of the currently executing method can be determined as the execution time of the first data processing thread. The execution time of the first data processing thread can be set as the execution time of the first data processing task N. iResource consumption data during execution. Among them, the first data processing task N... i The data volume of resource consumption data during execution exceeding the preset consumption threshold includes: the data volume of the execution time data of the first data processing thread exceeding the preset execution time threshold. In other words, when the data volume of the execution time data of the first data processing thread exceeds the preset consumption threshold, the first data processing task N can be... i This is designated as the second data processing task. The preset execution time threshold is the upper limit of the normal execution time of a method. For example, most methods take between 30 and 60 milliseconds to execute, while the preset execution time threshold can be 80 milliseconds.

[0127] In another implementation, the stack snapshot information includes the method call chain, which describes the path and hierarchical relationship of a series of methods calling each other in a specific order during task execution. Analyzing the execution information yields the first data processing task N. i Resource consumption data during execution includes: determining the first data processing task N based on the data processing statements contained in the execution information. i The first data processing thread is used in the first data processing thread pool; the number of calls to various methods in the method call chain included in the stack snapshot information of the first data processing thread is counted; the maximum number of calls among the counted calls is set as the first data processing task N. i Resource consumption data during execution; where the first data processing task N i The data volume of resource consumption data during execution exceeds the preset consumption threshold, including: the maximum number of calls exceeding the preset call count threshold. In other words, when the maximum number of calls exceeds the preset call count threshold, the first data processing task N can be... i This task is designated as the second data processing task. The preset call threshold is the upper limit for the number of times a method can be called within a normal range. It can be set based on experience or the call count of most methods over a past period.

[0128] In another implementation, the stack snapshot information includes the state of the corresponding data processing thread (such as congestion state, running state, waiting state, etc.). Analyzing the execution information yields the first data processing task N. i Resource consumption data during execution includes: determining the first data processing task N based on the data processing statements contained in the execution information. iThe first data processing thread is used in the first data processing thread pool; the state of the first data processing thread is obtained from the stack snapshot information of the first data processing thread; when the state of the first data processing thread is in the target state (which can be a congested state or a waiting state), the duration of the first data processing thread in the target state can be set as the first data processing task N. i Resource consumption data during execution; where the first data processing task N i The data volume of resource-consuming data during execution exceeding a preset consumption threshold includes: the duration of the first data processing thread in the target state exceeding a preset duration threshold (this could be due to thread blocking or waiting caused by resource contention). In other words, when the duration of the first data processing thread in the target state exceeds the preset duration threshold, the first data processing task N can be... i This is designated as the second data processing task. The preset duration threshold is the upper limit of the normal range for the duration of a data processing thread in the target state. For example, if performance testing reveals that when the duration of a data processing thread in a congested state exceeds 45 seconds, the response time of the data node begins to deteriorate significantly, then the preset duration threshold could be 40 seconds.

[0129] S44. Intervene in the execution process of the second data processing task.

[0130] In one embodiment, intervening in the execution of a second data processing task includes: stopping the execution of the second data processing task if it meets the task cancellation conditions; since the second data processing task can be divided into at least one subtask, and a subtask is typically executed by a data processing thread in a distributed database system, stopping the execution of the second data processing task includes: sending corresponding cancellation instructions to the data processing threads executing at least one subtask, respectively, to instruct them to cancel the execution of the corresponding subtask. The conditions for the second data processing task to meet the task cancellation conditions include any of the following:

[0131] 1) Enable the task cancellation switch in the distributed database system. In a distributed database system, a task cancellation switch can be configured to provide a flexible task management mechanism. When the task cancellation switch is enabled (which can be enabled by the distributed database system administrator), it means that the distributed database system allows the cancellation of the execution process of the second data processing task. This provides the distributed database system administrator with a macro-level control over task execution, ensuring system stability and the rational utilization of resources.

[0132] 2) The execution progress of the second data processing task has not reached the preset execution progress. For example, the second data processing task can be broken down into at least one subtask. The execution progress of the second data processing task can be determined by the ratio between the number of currently completed subtasks and the total number of subtasks. The preset execution progress can be a pre-set percentage (e.g., 0.9) that the subtasks of the second data processing task should complete at the current time. If the execution progress of the second data processing task has not reached the preset execution progress, it indicates that the second data processing task may have at least one of the following problems: the second data processing task itself has problems, such as low algorithm efficiency, logical errors, or data anomalies; or insufficient resource allocation, such as insufficient CPU or memory resources to support the efficient execution of the second data processing task. In this case, the execution of the second data processing task can be canceled to avoid a decrease in the overall performance of the distributed database system due to the inefficient execution of a single task.

[0133] 3) The second data processing task has no associated business process. Many data processing tasks are performed to support specific business processes and are closely related to them. If a second data processing task has no associated business process, it means that the second data processing task may be isolated and does not directly support the normal operation of the entire business. For example, in an e-commerce system, data processing tasks usually revolve around business processes such as order processing, inventory management, and customer service. If a data processing task collects the color information of all product images and classifies them by color, but the order processing, inventory management, and customer service business processes do not use this color-classified result, then this data processing task has no associated business process.

[0134] 4) The priority of the second data processing task is lower than the preset priority. In a distributed database system, different data processing tasks can be assigned different priorities to ensure that important or urgent data processing tasks are processed first. The preset priority is a pre-defined standard used to measure the importance of data processing tasks. When the priority of the second data processing task is lower than this preset priority, it indicates that the second data processing task is relatively less important or urgent. For example, in a company's data analysis system, the preset priority stipulates that tasks involving the calculation of key business indicators are high priority, while some routine data backup tasks are low priority. If the priority of the data backup task (the second data processing task) is lower than the preset high priority standard, it can be determined that the data backup task meets the task cancellation conditions, and the execution process of the data backup task can be stopped so that resources can be allocated to more important tasks.

[0135] In another embodiment, intervention in the execution process of the second data processing task includes: if the second data processing task meets the task interception conditions, instructing data nodes in the distributed database system to intercept the second data processing task. For example, the data processing statement of the second data processing task can be sent to the data nodes in the distributed database system, allowing the data nodes to add the data processing statement of the second data processing task to their managed task interception list. Thus, when the coordinating node detects that the data processing statement sent by the client is the data processing statement of the second data processing task, it can directly return an exception error to the client. The conditions for the second data processing task to meet the task interception conditions include:

[0136] 1) Enabling Task Interception in Distributed Database Systems. In distributed database systems, a task interception switch can be configured to provide a flexible task management mechanism. When the task interception switch is enabled (which can be done by the distributed database system administrator), it means that the distributed database system allows the interception of secondary data processing tasks. This provides the administrator with the ability to macroscopically control task execution, ensuring the stability and efficient operation of the system.

[0137] 2) The second data processing task has no related business process. For example, if the second data processing task is to collect data that is completely unrelated to the core business and has no other clear business purpose, this is the case. In this case, intercepting it can prevent system resources from being wasted on such meaningless tasks.

[0138] 3) The priority of the second data processing task is lower than the preset priority. This allows some lower-priority second data processing tasks to be intercepted, and resources to be allocated to more important tasks, thereby ensuring the overall performance of the system.

[0139] In another embodiment, intervention in the execution process of the second data processing task includes: sending execution log information of the second data processing task to the management terminal of the distributed database system. The execution log information records various key information during the execution of the second data processing task, providing a detailed record of its execution status. It can cover a series of operations, events, and related data experienced by the second data processing task from its inception to the current stage, comprehensively reflecting the status, process, and results of the second data processing task's execution. The execution log information can be used to instruct one or more of the following: adjustment strategies for the processing logic of the second data processing task, adjustment strategies for the data distribution of the distributed database system, and adjustment strategies for the business process of the second data processing task. In one implementation, the execution log information can be used to instruct the adjustment strategy for the processing logic of the second data processing task. For example, through the execution log information, the administrator can gain a deeper understanding of the performance of the processing logic of the second data processing task during actual execution, check whether the second data processing task processes data according to the expected algorithm and steps, and whether certain processing steps have taken too long, errors have occurred, or the processing results do not meet expectations. Based on the analysis of execution, if problems are found in the processing logic, such as inefficiency in a certain calculation step leading to slow overall task execution, or errors in the processing logic under certain specific data conditions, then a processing logic adjustment strategy can be formulated based on this information. For example, algorithms can be optimized, the order of data processing can be modified, or exception handling mechanisms can be added to improve the processing efficiency and accuracy of the second data processing task. In another implementation, execution log information can be used to indicate the data distribution adjustment strategy of the distributed database system. For example, execution log information can provide information about the usage of data in the distributed database system by the second data processing task, such as which data is frequently accessed, which data is accessed less frequently, and the distribution of data access across different data nodes. Administrators can determine whether the current data distribution is reasonable based on their understanding of data usage patterns. For example, if some data nodes are overloaded while others are underloaded, it may indicate an unbalanced data distribution. In this case, a data distribution adjustment strategy can be formulated based on the execution log information, such as migrating some data from high-load nodes to low-load nodes, or reorganizing the data storage method, to improve the overall performance and data processing efficiency of the system, ensuring that the distributed database system can more efficiently support the second data processing task.In another implementation, execution log information can be used to indicate business process adjustment strategies for the second data processing task. For example, if problems are found in the business process during the execution of the second data processing task, such as bottlenecks in the business process or poor coordination between different business links, the administrator can formulate business process adjustment strategies based on this information. For example, the business process can be re-examined and optimized, the execution order of the second data processing task can be adjusted, and unnecessary second data processing tasks can be reduced to make the business process smoother and more efficient, while also improving the overall operational efficiency and quality of the business.

[0140] Therefore, the task processing method proposed in this application, when locating the second data processing task with high resource consumption, can transform the problem of statistically analyzing task resource consumption into the problem of statistically analyzing the number of data processing tasks occupying data processing threads higher than a preset thread consumption threshold at the same time point, and the problem of detecting hot threads. Since the allocation and usage of system resources are dynamic, and the number of threads is easier to statistically analyze than resource consumption, and the detection of hot threads can also be efficiently filtered through specific tools and algorithms, the amount of data processing can be greatly reduced, and the accuracy and speed of task location can be improved. In addition, the task location method proposed in this application is more conducive to optimization at the system level. For example, by identifying which tasks consume too many thread resources and which are hot stacks, administrators can perform targeted optimizations, such as optimizing the execution code of hot stacks and adjusting business processes. This helps to improve the resource utilization and performance of the distributed database system, avoid the overall performance degradation caused by excessive resource consumption of individual tasks, and thus achieve the goal of enhancing the stability and availability of the distributed database system.

[0141] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0142] Figure 7 This is a schematic diagram of a task processing device provided in an embodiment of this application; the task processing device can be used to execute some or all of the steps in the foregoing method embodiments. Please refer to... Figure 7 The task processing device includes the following units: an acquisition unit 71 and a processing unit 72.

[0143] The acquisition unit 71 is used to acquire the execution information of the N first data processing tasks to be inspected in the distributed database system when the inspection time of the distributed database system arrives, where N is a positive integer;

[0144] The processing unit 72 is used to analyze the execution information to obtain the resource consumption data when each first data processing task is executed;

[0145] The processing unit 72 is also used to locate the second data processing task from the N first data processing tasks based on the resource consumption data when each first data processing task is executed, wherein the amount of resource consumption data when the second data processing task is executed is higher than a preset consumption threshold.

[0146] The processing unit 72 is also used to intervene in the execution process of the second data processing task.

[0147] In one embodiment, the distributed database system includes multiple data nodes, each data node running a data processing thread pool, and a data processing task in the distributed database system is assigned to the data processing thread pool on at least one data node for execution; the processing unit 72 is further configured to:

[0148] Call the acquisition unit 71 to obtain the operating load of each data node;

[0149] Based on the operating load of each data node, a first data node is determined from multiple data nodes. The operating load of the first data node is higher than a preset load threshold, and a first data processing thread pool is running on the first data node.

[0150] The data processing tasks executed in the first data processing thread pool are identified as the first data processing tasks to be inspected in the distributed database system.

[0151] In one embodiment, the first data processing thread pool contains at least one data processing thread, which is used to execute the data processing tasks assigned to the first data node; the processing unit 72 determines the data processing tasks executed in the first data processing thread pool as the first data processing task to be inspected in the distributed database system, including:

[0152] Retrieve the data processing record corresponding to each data processing thread in the first data processing thread pool. The data processing record includes the data processing statement executed by the corresponding data processing thread.

[0153] The data processing tasks represented by the data processing statements executed by each data processing thread in the first data processing thread pool are identified as the first data processing tasks to be inspected in the distributed database system.

[0154] In one embodiment, the first data processing thread pool contains at least one data processing thread, and each data processing thread in the first data processing thread pool corresponds to its own data processing record; the acquisition unit 71 acquires the execution information of N first data processing tasks to be inspected in the distributed database system, including:

[0155] Set the data processing records corresponding to each data processing thread in the first data processing thread pool as execution information.

[0156] In one embodiment, the execution information includes a data processing record corresponding to each data processing thread in the first data processing thread pool, and the data processing record includes the data processing statements executed by the corresponding data processing thread; any one of the N first data processing tasks is represented as the first data processing task N. i , where i is a positive integer less than or equal to N;

[0157] Processing unit 72 analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0158] Based on the data processing statements contained in the execution information, calculate the first data processing task N. i The number of data processing threads used in the first data processing thread pool;

[0159] Set the quantity to N for the first data processing task. i Resource consumption data during execution;

[0160] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the number of data exceeding the preset thread consumption threshold.

[0161] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. The data processing records include: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread; any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N;

[0162] Processing unit 72 analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0163] Based on the data processing statements contained in the execution information, determine the first data processing task N. i The first data processing thread occupied in the first data processing thread pool;

[0164] Obtain the resource utilization rate of the first data processing thread in the first data node from the stack snapshot information of the first data processing thread;

[0165] Set the resource utilization rate to the first data processing task N. i Resource consumption data during execution;

[0166] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: resource utilization rate exceeding the preset resource utilization rate threshold.

[0167] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. The data processing records include: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread. The stack snapshot information includes the method call chain. Any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N;

[0168] Processing unit 72 analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0169] Based on the data processing statements contained in the execution information, determine the first data processing task N. i The first data processing thread occupied in the first data processing thread pool;

[0170] The stack snapshot information of the first data processing thread includes the number of calls to various methods in the method call chain;

[0171] Set the maximum number of calls among the statistically obtained call counts as the first data processing task N. i Resource consumption data during execution;

[0172] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the maximum number of calls exceeding the preset call count threshold.

[0173] In one embodiment, the processing unit 72 intervenes in the execution process of the second data processing task, including:

[0174] If the second data processing task meets the task cancellation conditions, then the execution process of the second data processing task will be stopped.

[0175] The conditions for canceling the second data processing task include any of the following: the execution progress of the second data processing task has not reached the preset execution progress, the priority of the second data processing task is lower than the preset priority, the second data processing task has no associated business process, or the distributed database system has the task cancellation switch turned on.

[0176] In one embodiment, the second data processing task is divided into at least one subtask, and one subtask is executed by a data processing thread in the distributed database system; the processing unit 72 stops the execution of the second data processing task, including:

[0177] Send corresponding cancellation instructions to the data processing threads that are executing at least one subtask, respectively, to indicate that the corresponding subtasks should be cancelled.

[0178] In one embodiment, the processing unit 72 intervenes in the execution process of the second data processing task, including:

[0179] If the second data processing task meets the task interception conditions, then instruct the data nodes in the distributed database system to intercept the second data processing task.

[0180] The conditions for the second data processing task to meet the task interception criteria include: the second data processing task has no associated business process, the priority of the second data processing task is lower than the preset priority, and the task interception switch of the distributed database system is turned on.

[0181] In one embodiment, the processing unit 72 intervenes in the execution process of the second data processing task, including:

[0182] Send the execution log information of the second data processing task to the management terminal of the distributed database system. The execution log information is used to indicate one or more of the following: the processing logic adjustment strategy of the second data processing task, the data distribution adjustment strategy of the distributed database system, and the business process adjustment strategy of the second data processing task.

[0183] In one embodiment, the method for detecting the arrival of the inspection time includes at least one of the following:

[0184] When the preset inspection cycle is reached, the inspection time is determined.

[0185] When the frequency of task requests in the distributed database system exceeds a preset frequency threshold, the inspection time is determined.

[0186] When the resource utilization rate of a distributed database system exceeds a preset utilization rate threshold, the inspection time is determined.

[0187] When the task execution failure rate of a distributed database system exceeds a preset failure rate threshold, the inspection time is determined.

[0188] According to one embodiment of this application, Figure 7 The various units in the task processing device shown can be individually or entirely combined into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the task processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by multiple units working together. According to another embodiment of this application, a computer program (including program code) capable of performing the steps involved in the aforementioned method can be run on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM), to construct a device such as a computer. Figure 7 The task processing apparatus shown herein, and the task processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0189] The task processing scheme provided in this application embodiment can obtain the execution information of N first data processing tasks to be inspected in the distributed database system when the inspection time arrives. These N first data processing tasks are determined based on the inspection mechanism of the distributed database system. This mechanism allows for dynamic planning of the inspection scope, significantly improving the task inspection efficiency of the distributed database system. Statistical analysis of the execution information yields resource consumption data for each first data processing task. Based on this data, second data processing tasks are identified from the N first data processing tasks, where the resource consumption data exceeds a preset threshold. Through statistical analysis of the execution information, high-resource-consuming second data processing tasks in the distributed database system can be identified, thus pinpointing high-load data processing tasks that may cause load spikes and compromise customer SLAs. Intervention processing is applied to the execution process of these second data processing tasks. This intervention can promptly eliminate potential load risks associated with these tasks, improving the manageability of the distributed database system for these tasks and ultimately enhancing the stability and availability of the distributed database system.

[0190] Please see Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 8 As shown, the computer device 80 may include an input device 81, an output device 82, a processor 83, a memory 84, a network interface 85, and at least one communication bus 86. The processor 83 may be a Central Processing Unit (CPU). The processor may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. The PLDs may be field-programmable gate arrays (FPGAs), generic array logic (GALs), etc.

[0191] Memory 84 may include volatile memory, such as random-access memory (RAM); memory 84 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; memory 84 may be high-speed RAM or non-volatile memory, such as at least one disk drive. Optionally, memory 84 may also be at least one storage device located remotely from the aforementioned processor 83. Memory 84 may also include combinations of the above-mentioned types of memory. Figure 8 As shown, the memory 84, which is a computer-readable storage medium, may include an operating system, a network communication module, an object interface module, and a device control application.

[0192] The network interface 85 may include a standard wired interface or a wireless interface (such as a Wi-Fi interface) for providing data communication functions; the communication bus 86 is responsible for connecting various communication components; the input device 81 receives instructions from the object input to generate signal inputs related to the object settings and function control of the terminal device. In one embodiment, the input device 81 includes, but is not limited to, one or more of a touch panel, a physical keyboard or virtual keyboard, function keys, and a mouse; the output device 82 is used to output data information and may include a display screen or other display device; the processor 83 is the control center of the terminal device, connected to various parts of the entire terminal device by various interfaces and lines, and performs various functions by scheduling and running computer programs stored in the memory 84.

[0193] The processor 83 can be used to call computer programs in memory to perform the following operations:

[0194] When the inspection time of the distributed database system is reached, obtain the execution information of the N first data processing tasks to be inspected in the distributed database system, where N is a positive integer;

[0195] Analyze the execution information to obtain the resource consumption data when each first data processing task is executed;

[0196] Based on the resource consumption data when each first data processing task is executed, the second data processing task is located from N first data processing tasks. The amount of resource consumption data when the second data processing task is executed is higher than the preset consumption threshold.

[0197] Intervene in the execution process of the second data processing task.

[0198] In one embodiment, the distributed database system includes multiple data nodes, each data node running a data processing thread pool, and a data processing task in the distributed database system is assigned to the data processing thread pool on at least one data node for execution; the processor 83 is further configured to:

[0199] Obtain the operating load of each data node;

[0200] Based on the operating load of each data node, a first data node is determined from multiple data nodes. The operating load of the first data node is higher than a preset load threshold, and a first data processing thread pool is running on the first data node.

[0201] The data processing tasks executed in the first data processing thread pool are identified as the first data processing tasks to be inspected in the distributed database system.

[0202] In one embodiment, a first data processing thread pool contains at least one data processing thread, which is used to execute data processing tasks assigned to the first data node; the processor 83 determines the data processing tasks executed in the first data processing thread pool as the first data processing task to be inspected in the distributed database system, including:

[0203] Retrieve the data processing record corresponding to each data processing thread in the first data processing thread pool. The data processing record includes the data processing statement executed by the corresponding data processing thread.

[0204] The data processing tasks represented by the data processing statements executed by each data processing thread in the first data processing thread pool are identified as the first data processing tasks to be inspected in the distributed database system.

[0205] In one embodiment, the first data processing thread pool contains at least one data processing thread, and each data processing thread in the first data processing thread pool corresponds to its own data processing record; the processor 83 obtains the execution information of N first data processing tasks to be inspected in the distributed database system, including:

[0206] Set the data processing records corresponding to each data processing thread in the first data processing thread pool as execution information.

[0207] In one embodiment, the execution information includes a data processing record corresponding to each data processing thread in the first data processing thread pool, and the data processing record includes the data processing statements executed by the corresponding data processing thread; any one of the N first data processing tasks is represented as the first data processing task N. i , where i is a positive integer less than or equal to N;

[0208] Processor 83 analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0209] Based on the data processing statements contained in the execution information, calculate the first data processing task N. i The number of data processing threads used in the first data processing thread pool;

[0210] Set the quantity to N for the first data processing task. i Resource consumption data during execution;

[0211] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the number of data exceeding the preset thread consumption threshold.

[0212] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. The data processing records include: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread; any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N;

[0213] Processor 83 analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0214] Based on the data processing statements contained in the execution information, determine the first data processing task N. i The first data processing thread occupied in the first data processing thread pool;

[0215] Obtain the resource utilization rate of the first data processing thread in the first data node from the stack snapshot information of the first data processing thread;

[0216] Set the resource utilization rate to the first data processing task N. i Resource consumption data during execution;

[0217] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: resource utilization rate exceeding the preset resource utilization rate threshold.

[0218] In one embodiment, the execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. The data processing records include: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread. The stack snapshot information includes the method call chain. Any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N;

[0219] Processor 83 analyzes the execution information to obtain the first data processing task N. i Resource consumption data during execution includes:

[0220] Based on the data processing statements contained in the execution information, determine the first data processing task N. i The first data processing thread occupied in the first data processing thread pool;

[0221] The stack snapshot information of the first data processing thread includes the number of calls to various methods in the method call chain;

[0222] Set the maximum number of calls among the statistically obtained call counts as the first data processing task N. i Resource consumption data during execution;

[0223] Among them, the first data processing task N i The amount of resource consumption data during execution that exceeds the preset consumption threshold includes: the maximum number of calls exceeding the preset call count threshold.

[0224] In one embodiment, the processor 83 intervenes in the execution process of the second data processing task, including:

[0225] If the second data processing task meets the task cancellation conditions, then the execution process of the second data processing task will be stopped.

[0226] The conditions for canceling the second data processing task include any of the following: the execution progress of the second data processing task has not reached the preset execution progress, the priority of the second data processing task is lower than the preset priority, the second data processing task has no associated business process, and the distributed database system has the task cancellation switch turned on.

[0227] In one embodiment, the second data processing task is divided into at least one subtask, and one subtask is executed by a data processing thread in the distributed database system; the processor 83 stops the execution of the second data processing task, including:

[0228] Send corresponding cancellation instructions to the data processing threads that are executing at least one subtask, respectively, to indicate that the corresponding subtasks should be cancelled.

[0229] In one embodiment, the processor 83 intervenes in the execution process of the second data processing task, including:

[0230] If the second data processing task meets the task interception conditions, then instruct the data nodes in the distributed database system to intercept the second data processing task.

[0231] The conditions for the second data processing task to meet the task interception criteria include: the second data processing task has no associated business process, the priority of the second data processing task is lower than the preset priority, and the task interception switch of the distributed database system is turned on.

[0232] In one embodiment, the processor 83 intervenes in the execution process of the second data processing task, including:

[0233] Send the execution log information of the second data processing task to the management terminal of the distributed database system. The execution log information is used to indicate one or more of the following: the processing logic adjustment strategy of the second data processing task, the data distribution adjustment strategy of the distributed database system, and the business process adjustment strategy of the second data processing task.

[0234] In one embodiment, the method for detecting the arrival of the inspection time includes at least one of the following:

[0235] When the preset inspection cycle is reached, the inspection time is determined.

[0236] When the frequency of task requests in the distributed database system exceeds a preset frequency threshold, the inspection time is determined.

[0237] When the resource utilization rate of a distributed database system exceeds a preset utilization rate threshold, the inspection time is determined.

[0238] When the task execution failure rate of a distributed database system exceeds a preset failure rate threshold, the inspection time is determined.

[0239] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the task processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0240] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0241] Furthermore, it should be noted that embodiments of this application also provide a computer-readable storage medium storing a computer program for the aforementioned task processing method. This computer program includes computer instructions. When one or more processors load and execute these computer instructions, the task processing method described in the embodiments can be implemented, and will not be repeated here. The beneficial effects of using the same method will also not be repeated here. It is understood that the computer instructions can be deployed on one or more computer devices capable of communicating with each other for execution.

[0242] The aforementioned computer-readable storage medium can be an internal storage unit of the task processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0243] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method provided in one aspect of the embodiments of this application.

[0244] In one aspect of this application, another computer program product is provided, which includes a computer program or computer instructions that, when executed by a processor, implement the steps of the task processing method provided in the embodiments of this application.

[0245] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A task processing method, characterized in that, The method is applied to a distributed database system, which is used to perform data processing tasks; the method includes: When the inspection time of the distributed database system is reached, the execution information of the N first data processing tasks to be inspected in the distributed database system is obtained, where N is a positive integer; The execution information is analyzed to obtain the resource consumption data when each of the first data processing tasks is executed; Based on the resource consumption data when each of the first data processing tasks is executed, a second data processing task is located from the N first data processing tasks, wherein the amount of resource consumption data when the second data processing task is executed is higher than a preset consumption threshold. Intervene in the execution process of the second data processing task.

2. The method as described in claim 1, characterized in that, The distributed database system contains multiple data nodes, and each data node runs a data processing thread pool. A data processing task in the distributed database system is assigned to at least one data processing thread pool on the data node for execution. The method further includes: Obtain the operating load of each of the data nodes; Based on the operating load of each of the data nodes, a first data node is determined from the plurality of data nodes, wherein the operating load of the first data node is higher than a preset load threshold, and a first data processing thread pool is running on the first data node. The data processing task executed in the first data processing thread pool is identified as the first data processing task to be inspected in the distributed database system.

3. The method as described in claim 2, characterized in that, The first data processing thread pool contains at least one data processing thread, and the data processing thread in the first data processing thread pool is used to execute the data processing task assigned to the first data node; The step of determining the data processing task executed in the first data processing thread pool as the first data processing task to be inspected in the distributed database system includes: Obtain the data processing record corresponding to each data processing thread in the first data processing thread pool, wherein the data processing record includes the data processing statement executed by the corresponding data processing thread; The data processing tasks represented by the data processing statements executed by each of the data processing threads in the first data processing thread pool are determined as the first data processing task to be inspected in the distributed database system.

4. The method as described in claim 2 or 3, characterized in that, The first data processing thread pool contains at least one data processing thread, and each data processing thread in the first data processing thread pool corresponds to its own data processing record; obtaining the execution information of the N first data processing tasks to be inspected in the distributed database system includes: Set the data processing records corresponding to each data processing thread in the first data processing thread pool as the execution information.

5. The method as described in claim 4, characterized in that, The execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool, and the data processing records include data processing statements executed by the corresponding data processing thread; any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N; The execution information is analyzed to obtain the first data processing task N. i Resource consumption data during execution includes: Based on the data processing statements contained in the execution information, the first data processing task N is statistically analyzed. i The number of data processing threads occupied in the first data processing thread pool; Set the quantity as N for the first data processing task. i Resource consumption data during execution; Wherein, the first data processing task N i The amount of resource consumption data during execution that exceeds a preset consumption threshold includes: the amount exceeding a preset thread consumption threshold.

6. The method as described in claim 4, characterized in that, The execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. Each data processing record includes: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread. Any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N; The execution information is analyzed to obtain the first data processing task N. i Resource consumption data during execution includes: Based on the data processing statements contained in the execution information, the first data processing task N is determined. i The first data processing thread occupied in the first data processing thread pool; Obtain the resource utilization rate of the first data processing thread in the first data node from the stack snapshot information of the first data processing thread; Set the resource utilization rate as the first data processing task N. i Resource consumption data during execution; Wherein, the first data processing task N i The amount of resource consumption data during execution that exceeds a preset consumption threshold includes: the resource utilization rate exceeding a preset resource utilization rate threshold.

7. The method as described in claim 4, characterized in that, The execution information includes data processing records corresponding to each data processing thread in the first data processing thread pool. Each data processing record includes: the data processing statements executed by the corresponding data processing thread and the stack snapshot information of the corresponding data processing thread, the stack snapshot information including the method call chain; any one of the N first data processing tasks is represented as first data processing task N. i , where i is a positive integer less than or equal to N; The execution information is analyzed to obtain the first data processing task N. i Resource consumption data during execution includes: Based on the data processing statements contained in the execution information, the first data processing task N is determined. i The first data processing thread occupied in the first data processing thread pool; Count the number of calls to various methods in the method call chain included in the stack snapshot information of the first data processing thread; Set the maximum number of calls among the statistically obtained call counts as the first data processing task N. i Resource consumption data during execution; Wherein, the first data processing task N i The amount of resource consumption data during execution that exceeds a preset consumption threshold includes: the maximum number of calls exceeding a preset call count threshold.

8. The method as described in claim 1, characterized in that, The intervention process for the execution of the second data processing task includes: If the second data processing task meets the task cancellation conditions, then the execution process of the second data processing task is stopped; The second data processing task meets the task cancellation conditions in any of the following ways: the execution progress of the second data processing task has not reached the preset execution progress, the priority of the second data processing task is lower than the preset priority, the second data processing task has no associated business process, and the distributed database system turns on the task cancellation switch.

9. The method as described in claim 8, characterized in that, The second data processing task is broken down into at least one subtask, and one of the subtasks is executed by a data processing thread in the distributed database system; The process of stopping the execution of the second data processing task includes: Send corresponding cancellation instructions to the data processing threads executing the at least one subtask, respectively, to instruct them to cancel the execution of the corresponding subtask.

10. The method as described in claim 1, characterized in that, The intervention process for the execution of the second data processing task includes: If the second data processing task meets the task interception conditions, then the data nodes in the distributed database system are instructed to intercept the second data processing task. The second data processing task meets the task interception conditions as follows: the second data processing task has no associated business process, the priority of the second data processing task is lower than the preset priority, and the distributed database system turns on the task interception switch.

11. The method as described in claim 1, characterized in that, The intervention process for the execution of the second data processing task includes: The execution log information of the second data processing task is sent to the management terminal of the distributed database system. The execution log information is used to indicate one or more of the following: the processing logic adjustment strategy of the second data processing task, the data distribution adjustment strategy of the distributed database system, and the business process adjustment strategy of the second data processing task.

12. The method as described in claim 1, characterized in that, The detection method for reaching the inspection time includes at least one of the following: When the preset inspection cycle is reached, the inspection time is determined. When the task request frequency of the distributed database system is higher than a preset frequency threshold, the inspection time is determined to be reached. When the resource utilization rate of the distributed database system exceeds a preset utilization rate threshold, the inspection time is determined to be reached. When the task execution failure rate of the distributed database system exceeds a preset failure rate threshold, the inspection time is determined.

13. A task processing device, characterized in that, The apparatus is applied to a distributed database system, which is used to perform data processing tasks; the apparatus includes: The acquisition unit is used to acquire the execution information of N first data processing tasks to be inspected in the distributed database system when the inspection time of the distributed database system arrives, where N is a positive integer; The processing unit is used to analyze the execution information to obtain resource consumption data when each of the first data processing tasks is executed; The processing unit is further configured to locate a second data processing task from the N first data processing tasks based on the resource consumption data when each first data processing task is executed, wherein the amount of resource consumption data when the second data processing task is executed is higher than a preset consumption threshold. The processing unit is also used to intervene in the execution process of the second data processing task.

14. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the task processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including computer instructions, which, when executed by a processor, perform the task processing method according to any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions that, when executed by a processor, implement the steps of the task processing method as described in any one of claims 1 to 12.

Images