Task scheduling method and device, equipment, medium and product

By dividing the input buffer and monitoring resource utilization in the distributed interactive analysis engine, and adjusting the task parallelism, the performance problem caused by cross-node data interaction was solved, improving system performance and reducing costs.

CN113971082BActive Publication Date: 2025-12-30BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111241693.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-25
Publication Date
2025-12-30
Estimated Expiration
2041-10-25

AI Technical Summary

Technical Problem

In a distributed interactive analytics engine, cross-node data interaction leads to network I/O, disk I/O, and CPU overhead, affecting system performance and increasing user costs.

Method used

By partitioning the input buffer in the target computing node, monitoring the resource utilization status, and adjusting the parallelism of tasks in other computing nodes according to the preset scheduling strategy, the additional overhead of cross-node data interaction can be reduced.

Benefits of technology

It effectively reduces the cross-node data interaction overhead when computing nodes execute tasks in parallel, improves the overall performance of the interactive analysis engine, and reduces the hardware and network costs for users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113971082B_ABST
    Figure CN113971082B_ABST
Patent Text Reader

Abstract

The present disclosure provides a task scheduling method and device, equipment, medium and product, relates to the field of data processing, in particular to the field of cloud computing. The specific implementation scheme is: obtaining a resource utilization state of an input buffer in a target computing node; wherein the input buffer is a data storage area pre-divided in the target computing node, used for storing data interacting with other computing nodes; according to the resource utilization state of the input buffer, and according to a preset scheduling strategy, adjusting the parallelism of a task executed in other computing nodes. The scheme of the present disclosure avoids the additional overhead generated by cross-node data interaction, and improves the efficiency of cross-node data interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more particularly to the field of cloud computing, specifically to a task scheduling method, apparatus, device, medium, and product. Background Technology

[0002] In the application of distributed interactive analytics engines, cross-node data interaction is inevitable. Cross-node data interaction incurs network I / O, disk I / O, and CPU overhead, leading to additional memory overhead. These overheads directly impact the performance of the system within the engine, such as query performance. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, medium, and product for task scheduling.

[0004] According to one aspect of this disclosure, a task scheduling method is provided, comprising:

[0005] Obtain the resource utilization status of the input buffer in the target computing node; wherein, the input buffer is a data storage area pre-divided in the target computing node for storing data interacting with other computing nodes;

[0006] Based on the resource utilization status of the input buffer, the parallelism of tasks executed on other computing nodes is adjusted according to a preset scheduling strategy.

[0007] According to another aspect of this disclosure, a task scheduling apparatus is provided, comprising:

[0008] The buffer resource determination module is used to obtain the resource utilization status of the input buffer in the target computing node; wherein, the input buffer is a data storage area pre-divided in the target computing node for storing data interacting with other computing nodes;

[0009] The task adjustment module is used to adjust the parallelism of tasks executed on other computing nodes according to the resource utilization status of the input buffer and a preset scheduling strategy.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0011] At least one processor; and

[0012] A memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the task scheduling method described in any embodiment of this disclosure.

[0014] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a task scheduling method according to any embodiment of this disclosure.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the task scheduling method according to any embodiment of this disclosure.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0018] Figure 1 This is a schematic diagram of a task scheduling method according to an embodiment of the present disclosure;

[0019] Figure 2 This is a schematic diagram of another task scheduling method according to an embodiment of the present disclosure;

[0020] Figure 3 This is a schematic diagram illustrating the adjustment of the input buffer, output buffer, and task concurrency according to an embodiment of this disclosure;

[0021] Figure 4 This is a schematic diagram of another task scheduling method according to an embodiment of the present disclosure;

[0022] Figure 5 This is a schematic diagram of another task scheduling method according to an embodiment of the present disclosure;

[0023] Figure 6 This is a schematic diagram of the structure of a task scheduling device according to an embodiment of the present disclosure;

[0024] Figure 7 This is a block diagram of an electronic device used to implement the task scheduling method of the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1 This is a schematic diagram of a task scheduling method according to an embodiment of the present disclosure. This embodiment can be applied to situations where the efficiency of cross-node data interaction in a distributed system is improved. The method can be executed by a task scheduling device, which can be implemented by software and / or hardware and integrated into an electronic device. The electronic device involved in this embodiment can be a device with communication and computing capabilities, such as a local server.

[0027] In another optional implementation of this embodiment, the execution entity of the method is a scheduling node in a distributed interactive analysis engine. The distributed interactive analysis engine includes a scheduling node and at least two computing nodes, which are used to execute tasks in the distributed interactive analysis engine in parallel.

[0028] Distributed interactive analytics engines refer to interactive analytics engines deployed across multiple servers, allowing different users to perform online analysis and query operations. For example, interactive analytics engines are typically used in scenarios such as generating dynamic statistical reports, ad hoc queries, BI systems, and data visualization. These application scenarios are characterized by: sensitivity to query speed, high concurrency (shared by multiple business departments and numerous data analysts), high tolerance for query errors, large fluctuations in resource requirements (significant load fluctuations), and high workload during peak working hours. For instance, the online query performance of interactive analytics engines is the most widely used feature and also generates the most cross-node data interactions.

[0029] Deploying an interactive analytics engine across multiple servers in a distributed system inevitably leads to a significant increase in the volume and frequency of cross-node interaction data generated by users. Furthermore, cross-node data exchange incurs network I / O, disk I / O, and CPU overhead. Even if downstream nodes cannot process the interacting data immediately, further processing is often required, leading to the caching of cross-node interaction data in memory, which further increases memory overhead. These overheads directly impact the performance of the distributed system, especially in distributed interactive analytics engines, where cross-node data interaction directly affects user experience and increases user costs.

[0030] The distributed interactive analytics engine includes at least one scheduling node and at least two computing nodes. The scheduling node manages the scheduling of tasks within the computing nodes; for example, it manages the data slices that each task within a computing node needs to process. Upon receiving a user-issued instruction, the interactive analytics engine generates an execution plan based on that instruction and distributes the plan for distributed execution. The entire distributed execution plan can be divided into multiple execution tasks, which are then distributed across the computing nodes in the distributed server cluster to achieve parallel computation across multiple computing nodes. For example, the entire distributed execution plan can be divided into one or more stages. A stage represents a phase division within the distributed execution plan of the interactive analytics engine. A stage contains multiple tasks to complete the computational goals of that stage. These tasks are distributed across various computing nodes in the cluster, achieving parallel computation and improving engine performance and data throughput. Computing nodes execute tasks sequentially according to their respective stages. For example, all computing nodes first execute the tasks included in stage 1 in parallel, then execute the tasks included in stage 2 in parallel. In this disclosure, the scheduled tasks refer to tasks within the same stage. For example, scheduling nodes and computing nodes can be deployed on the same server or different servers in a distributed cluster. For instance, each server may serve as a scheduling node or a computing node. This disclosure does not limit the specific deployment.

[0031] The embodiments disclosed herein can effectively reduce the additional overhead caused by cross-compute node data interaction when compute nodes execute tasks in parallel, improve the overall performance of the interactive analysis engine, and reduce the hardware and network costs for users to use the engine.

[0032] For details, please refer to Figure 1 The method specifically includes the following:

[0033] S110. Obtain the resource utilization status of the input buffer in the target computing node; wherein, the input buffer is a data storage area pre-divided in the target computing node, used to store data interacting with other computing nodes.

[0034] In this system, the target compute node is any compute node in the distributed cluster. An input buffer is pre-allocated in the memory of each compute node to store interaction data with other compute nodes. Specifically, the input buffer stores the interaction data received by the target compute node from other compute nodes. This dedicated input buffer allows for continuous monitoring of the data interaction between the target compute node and other compute nodes, and also reduces additional memory overhead.

[0035] Resource utilization status refers to the usage of hardware resources in the input buffer, specifically the usage of memory space. Since the additional overhead from cross-node data interaction is mostly due to network consumption, resource utilization status can include not only memory usage but also CPU and network status. However, regardless of the cause of the additional overhead from cross-node data interaction, the final manifestation is insufficient remaining buffer memory. Therefore, in this embodiment, resource utilization status is memory utilization status, which can be characterized using metrics such as memory utilization rate.

[0036] For example, in a distributed interactive engine, a memory region is pre-allocated in the memory of each compute node as an input buffer to store interactive data received by that compute node from other compute nodes. The scheduling node monitors the resource usage of the input buffer in each compute node, such as memory utilization.

[0037] S120. Adjust the parallelism of tasks executed on other computing nodes according to the resource utilization status of the input buffer and the preset scheduling strategy.

[0038] The parallelism of a task refers to the maximum number of tasks that are being executed in parallel on a computing node. For example, the parallelism refers to the maximum amount of data that is being processed in parallel by a task, or the maximum number of tasks that are being executed in parallel. This parallelism reflects the speed at which a computing node generates data to be consumed. For example, when the parallelism of a task is reduced by decreasing the number of tasks being executed in parallel, the amount of data generated by the task is reduced.

[0039] The preset scheduling strategy refers to the strategic relationship between the resource utilization status of the buffer and the adjustment of the parallelism of the executed tasks. This relationship can be set according to the engine's performance and actual usage. For example, a mapping relationship is established between different resource utilization statuses and the parallelism processing methods of the executed tasks. When the resource utilization status is detected to reach any of the conditions in the mapping relationship, the parallelism processing method of the executed tasks is determined according to the correlation, and the parallelism of the tasks executed in the computing nodes is adjusted according to this processing method.

[0040] Specifically, when the scheduling node monitors the resource utilization status of the input buffers of each computing node, if it determines that the resource utilization of the input buffer in the target computing node is insufficient, such as when the memory utilization status reaches the preset limit, it means that the speed at which the target computing node consumes data for executing tasks is less than the speed at which the input buffer receives data, resulting in the data received from other computing nodes piling up in the input buffer and not being consumed.

[0041] To address the aforementioned issues, this disclosure adjusts the parallelism of tasks executed on other computing nodes to promptly manage data buildup in the input buffer, thereby preventing data overflows that could lead to additional memory consumption and impact engine performance. For example, when the input buffer's resources are underutilized, the parallelism of tasks executed on other computing nodes can be adjusted, thereby reducing the amount of data sent from other computing nodes to the target computing node.

[0042] The solution in this embodiment monitors the resource utilization status of the data input buffer in the computing node used to store data for interaction with other computing nodes, thereby monitoring cross-node data interaction data, avoiding the additional overhead caused by cross-node data interaction, and improving the efficiency of cross-node data interaction.

[0043] Figure 2 This is a schematic diagram of another task scheduling method according to an embodiment of the present disclosure. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the task scheduling methods include the following:

[0044] S210. Obtain the resource utilization status of the input buffer in the target computing node; wherein, the input buffer is used to store the interactive data sent by other computing nodes received by the target computing node.

[0045] The input buffer stores cross-node interaction data sent from other computing nodes to the target computing node; in other words, the interaction data in the input buffer is data awaiting consumption and processing by the target computing node. The resource utilization status of the input buffer represents the amount of data waiting to be consumed and processed by the target computing node. By monitoring the resource utilization status of the input buffer in the target computing node, the cross-node data interaction can be monitored.

[0046] Specifically, an input buffer is pre-allocated within the memory region of the target computing node. The size of the input buffer can be determined based on the normal data volume of interactive data from other computing nodes that the target computing node needs to process; that is, the region is allocated according to actual conditions, and the size of the buffer is not limited. During the operation of the distributed interactive analysis engine, the scheduling node associated with the target computing node monitors the resource utilization status of the input buffer to monitor the efficiency of the target computing node in processing cross-node interactive data. For example, in the above example, the resource utilization status can be memory utilization status, such as memory utilization rate.

[0047] S220. When a resource shortage is detected based on the resource utilization status of the input buffer, the upstream computing node of the target computing node is determined.

[0048] In this context, "resource insufficiency" refers to a situation where the hardware resources of the input buffer are insufficient, such as insufficient memory, leading to an impending overflow of cross-node interaction data stored within it. Resource insufficiency is generally caused by resource waste due to differences in processing capabilities between upstream and downstream nodes; for example, excessive upstream load may cause the input buffer to occupy resources for an extended period. An upstream computing node is a computing node that sends interaction data to the target computing node. According to the data flow direction, data from the upstream computing node flows to the target computing node, meaning the target computing node needs to process the data from the upstream computing nodes. For example, there is at least one upstream computing node; that is, all computing nodes that need to send interaction data to the target computing node are identified as upstream computing nodes.

[0049] Specifically, when the scheduling node monitors the resource utilization status of the input buffer, if it finds that the input buffer resources are insufficient, it determines the computing node that sends the interactive data to the target computing node as the upstream computing node based on the flow of interactive data. For example, the determination of upstream computing nodes can also be achieved by pre-establishing a mapping relationship. That is, when issuing execution tasks to various computing nodes, the computing nodes that have data interaction with the target computing node are determined based on the task execution status, and the nodes that need to send data to the target computing node are identified as the upstream computing nodes of the target computing node. A mapping relationship is then established so that the upstream computing node can be directly determined subsequently based on this mapping relationship.

[0050] In another optional implementation of this embodiment, the resource utilization status includes the resource utilization rate, and the resource insufficiency status means that the resource utilization rate of the input buffer is greater than a first preset resource utilization threshold.

[0051] Specifically, resource utilization status is used to characterize the resource usage of the input buffer, and can be represented by resource utilization rate, which is the ratio of the currently used resource value to the total resource value of the input buffer. For example, resources can refer to memory, so the resource utilization rate is the memory utilization rate, which is the ratio of the currently used memory of the input buffer to the total memory of the input buffer to represent the resource utilization status of the input buffer.

[0052] A resource shortage state refers to a situation where the remaining resources in the input buffer are insufficient. When characterized by resource utilization, this is determined when the resource utilization rate of the input buffer exceeds a first preset resource utilization threshold, indicating a resource shortage state. For example, when resource utilization is based on memory utilization, the scheduling node monitors the memory utilization rate of the input buffer in the target computing node. When the memory utilization rate exceeds the first preset resource utilization threshold, it indicates that the input buffer memory is about to fill up, posing a risk of overflow, and thus a resource shortage state is identified. The first preset resource utilization threshold can be set according to actual conditions and is not limited here; for example, it can be set to 95%, meaning that when the memory usage of the input buffer exceeds 95%, a resource shortage state is determined.

[0053] By characterizing resource utilization rate, we can accurately determine resource shortage status digitally, improve the accuracy of determining the data storage status of buffer interaction, thereby improving the efficiency of cross-node data interaction and avoiding additional resource overhead caused by insufficient buffer resources, which would lead to resource waste.

[0054] S230, Control the parallelism of data written from upstream computing nodes to target computing nodes.

[0055] The parallelism of writing data refers to the maximum number of data written in parallel, which is the maximum value of data sent in parallel from the upstream computing node to the target computing node.

[0056] Specifically, since data generated in the upstream computing node needs to be sent to the target computing node, the target computing node stores the interactive data sent from the upstream computing node in an input buffer. The input buffer then sends this data to the various tasks within the target computing node for processing. If the input buffer of the target computing node is in a resource-insufficient state, it means that the speed at which the tasks within the target computing node consume interactive data is less than the speed at which the input buffer receives interactive data. Because the consumption speed of interactive data by the tasks within the target computing node is limited, it is necessary to control the speed at which the input buffer receives interactive data to alleviate the problem of insufficient input buffer resources and avoid the additional memory overhead caused by input buffer memory overflow. The speed at which the input buffer receives interactive data largely depends on the speed at which the upstream computing node sends data to the target computing node. Therefore, by controlling the parallelism of the upstream node writing data to the target computing node, the speed at which the input buffer receives interactive data can be controlled, thereby controlling the resource utilization status of the input buffer in the target computing node.

[0057] For example, if the target compute node has at least two upstream compute nodes, the parallelism of the original write data between each upstream compute node and the target compute node is adjusted. For instance, the upstream compute node with the highest original write data parallelism is determined as the target upstream compute node, and the parallelism of the data written from the target upstream compute node to the target compute node is adjusted. The specific adjustment range of the parallelism can be determined based on the specific value of resource utilization or based on empirical values.

[0058] For example, such as Figure 3 The diagram shows the adjustment of the input buffer, output buffer, and task concurrency. Figure 3 The distributed system shown has one scheduling node and two compute nodes. The scheduling node controls the execution of tasks in compute node 1 and compute node 2. Compute nodes 1 and 2 are running three tasks in stage 1: task1, task2, and task3. Each compute node 1 and compute node 2 has an input buffer and an output buffer. The output buffer of compute node 1 stores the interactive data generated by task1 that needs to be sent to compute node 2. After receiving this interactive data, compute node 2 stores it in its input buffer, waiting for task2 and task3 to process it. Similarly, the output buffer of compute node 2 stores the interactive data generated by task2 and task3 that needs to be sent to compute node 1. After receiving this interactive data, compute node 1 stores it in its input buffer, waiting for task1 to process it.

[0059] When the scheduling node detects that the resource utilization of the input buffer in compute node 1 is greater than the first preset resource utilization threshold, it determines that the upstream compute node of compute node 1 is compute node 2, and adjusts the parallelism of compute node 2 writing data to compute node 1. This is reflected in... Figure 3 The solid line connection between the output buffer of compute node 2 and the input buffer of compute node 1 is changed to a dashed line connection. This reduces the maximum number of data items that compute node 2 can send to compute node 1 in parallel. Once task 1 has consumed the data in the input buffer and the input buffer resources are sufficient, the parallelism of data writing from compute node 2 to compute node 1 is readjusted. For example, when the resource utilization of the input buffer reaches a third preset resource utilization threshold, the parallelism of data writing from the upstream compute node to the target compute node is restored.

[0060] In another optional implementation of this embodiment, S230 includes:

[0061] Adjust the network connection parallelism for network transmission between the output buffer in the upstream computing node and the input buffer in the target computing node; wherein, the output buffer in the upstream computing node is a data storage area pre-divided in the upstream computing node for storing interactive data to be sent from the upstream computing node to other computing nodes; the adjusted network connection parallelism is less than the original network connection parallelism.

[0062] The additional overhead caused by insufficient utilization of input buffer resources is mostly due to network consumption. Since cross-node data interaction between upstream compute nodes and downstream target compute nodes is sent from the output buffer of the upstream compute node to the input buffer of the target compute node, the speed at which the input buffer receives interactive data can be controlled by controlling the parallelism of data writing from the upstream node to the target compute node. This can be achieved by reducing the network connection parallelism of network transmission between the output buffer in the upstream compute node and the input buffer in the target compute node, thereby controlling the speed at which the input buffer receives interactive data and ultimately reducing the resource utilization of the target buffer.

[0063] By controlling the parallelism of network connections in network transmission, the parallelism of data writing from upstream computing nodes to target computing nodes is controlled. This addresses the potential resource waste caused by cross-node data interaction by addressing its root causes, thereby preventing prolonged buffer resource occupation during periods of significant load fluctuation and improving the efficiency of cross-node data interaction.

[0064] The solution in this embodiment monitors the utilization status of the input buffer resources in the computing node to monitor the efficiency of the computing node in receiving cross-node interactive data. Based on the monitoring results, the parallelism of the data sending tasks of the upstream computing node is adjusted, and the input buffer's reception of interactive data is adjusted in a timely manner. This avoids the phenomenon that the input buffer occupies resources for a long time due to the difference in data processing capabilities between upstream and downstream nodes, which can lead to data processing stopping and additional memory consumption. This improves the efficiency of cross-node data interaction and increases the fairness of network resource sharing.

[0065] Figure 4 This is a schematic diagram of another task scheduling method according to an embodiment of the present disclosure. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 4 As shown, the task scheduling methods include the following:

[0066] S410. Obtain the resource utilization status of the output buffer in the target computing node; wherein, the output buffer is a data storage area pre-divided in the target computing node, used to store data interacting with other computing nodes.

[0067] In this system, the target compute node is any compute node in the distributed cluster. An output buffer is pre-allocated in the memory of each compute node to store data exchanged with other compute nodes. Specifically, the output buffer stores the data to be sent from the target compute node to other compute nodes. This dedicated output buffer allows for continuous monitoring of data exchange between the target compute node and other compute nodes, and also reduces additional memory overhead.

[0068] For example, in a distributed interactive engine, two memory regions are pre-allocated in the memory of each compute node: one serves as an input buffer to store interactive data received by that compute node from other compute nodes; the other serves as an output buffer to store interactive data to be sent by that compute node to other compute nodes. The scheduling node monitors the resource usage of the output and input buffers in each compute node, such as memory utilization.

[0069] S420. Adjust the parallelism of the tasks executed in the target computing node according to the resource utilization status of the output buffer and the preset scheduling strategy.

[0070] Specifically, when the scheduling node monitors the resource utilization status of the output buffers of each computing node, if it determines that the resource utilization of the output buffer of the target computing node is insufficient, it means that the data generated by the task execution in the target computing node is greater than the data transmission rate of the output buffer, resulting in the data that needs to be sent to other computing nodes piling up in the output buffer and not being consumed.

[0071] To address the aforementioned issues, this disclosure adjusts the parallelism of tasks executed on the target computing node to promptly manage data buildup in the output buffer, thereby preventing data overflows that could lead to additional memory consumption and impact engine performance. For example, when the output buffer's resources are underutilized, the parallelism of tasks executed on the target computing node can be adjusted. Adjusting the parallelism of these tasks reduces the amount of data generated on the target computing node that needs to be sent to other computing processes.

[0072] The solution in this embodiment monitors the resource utilization status of the output buffer in the computing node used to store data interacting with other computing nodes, thereby monitoring cross-node data interaction data, avoiding the additional overhead caused by cross-node data interaction, and improving the efficiency of cross-node data interaction.

[0073] Figure 5 This is a schematic diagram of another task scheduling method according to an embodiment of the present disclosure. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 5 As shown, the task scheduling methods include the following:

[0074] S510. Obtain the resource utilization status of the output buffer in the target computing node; wherein, the output buffer is a data storage area pre-divided in the target computing node, used to store data interacting with other computing nodes.

[0075] The output buffer stores cross-node interaction data that the target compute node needs to send to other compute nodes. In other words, the interaction data in the output buffer is data generated by the target compute node and awaiting transmission to other compute nodes for consumption. The resource utilization status of the output buffer represents the amount of data consumed by the target compute node. By monitoring the resource utilization status of the output buffer in the target compute node, the cross-node data interaction can be monitored.

[0076] Specifically, an output buffer is pre-allocated within the memory region of the target computing node. The size of the output buffer can be determined based on the speed and amount of cross-node interactive data generated by tasks executed on the target computing node; that is, the region is allocated according to actual conditions, and the size of the buffer is not limited. During the operation of the distributed interactive analysis engine, the scheduling node associated with the target computing node monitors the resource utilization status of the output buffer to monitor the efficiency of the target computing node in generating cross-node interactive data. For example, in the above example, the resource utilization status can be memory utilization status, such as memory utilization rate.

[0077] S520. When a resource shortage is detected based on the resource utilization status of the output buffer, the execution information of the task being executed in the target computing node is determined.

[0078] The "resource insufficiency" state refers to insufficient hardware resources in the output buffer, such as insufficient memory, leading to an impending overflow of cross-node interaction data stored within it. This state is generally caused by resource waste due to differences in upstream and downstream processing capabilities; for example, excessive downstream load may cause the output buffer to occupy resources for an extended period. Execution information includes details that distinguish the current task's execution status, such as the task's start time, the amount of data processed, and resource utilization.

[0079] Specifically, when the scheduling node monitors the resource utilization status of the input buffer, if it detects insufficient resources in the output buffer, it obtains the execution information of the tasks currently being executed on the target computing node. For example, if the target computing node's output buffer memory is insufficient, it indicates that an output buffer resource shortage has been detected. This means that the tasks executing on the target computing node are generating too much interactive data that needs to be sent to other computing nodes, leading to insufficient storage memory in the target computing node's output buffer. If the tasks executing on the target computing node continue to generate interactive data at the current rate, the output buffer will become full, causing data processing to stop and occupying additional memory space on the target computing node, resulting in resource waste and impacting engine performance. Therefore, when the output buffer resource is insufficient, the scheduling node first determines the execution information of the tasks currently executing on the target computing node, and then processes the tasks accordingly to reduce the amount of interactive data they generate.

[0080] In another optional implementation of this embodiment, the resource utilization status includes the resource utilization rate, and the resource insufficiency status means that the resource utilization rate of the output buffer is greater than the second preset resource utilization threshold.

[0081] Specifically, resource utilization status is used to characterize the resource usage of the output buffer. It can be represented by resource utilization rate, which is the ratio of the currently used resource value to the total resource value of the output buffer. For example, resources can refer to memory, so the resource utilization rate is the memory utilization rate, which is the ratio of the currently used memory of the output buffer to the total memory of the output buffer.

[0082] A resource shortage state refers to insufficient remaining resources in the output buffer. Characterized by resource utilization, this is determined when the output buffer's resource utilization exceeds a second preset resource utilization threshold. For example, when resource utilization is based on memory utilization, the scheduling node monitors the memory utilization of the output buffer in the target computing node. When the memory utilization exceeds the second preset resource utilization threshold, it indicates that the output buffer is about to fill up, posing an overflow risk, and thus a resource shortage state is identified. The second preset resource utilization threshold can be set according to actual conditions and is not limited here. For example, it can be set to 95%, meaning that when the output buffer's memory usage exceeds 95%, a resource shortage state is determined. The specific values ​​of the first and second preset resource utilization thresholds can be the same or different, determined based on the actual execution of related tasks, and are not limited here.

[0083] S530. Adjust the parallelism of the tasks executed in the target computing node according to the execution information.

[0084] Since the interactive data in the output buffer of the target computing node originates from the tasks executed within the target computing node, it is necessary to reduce the interactive data generated by the tasks executed within the target computing node when output buffer resources are insufficient. In this embodiment, the interactive data generated by the tasks executed within the target computing node is reduced by adjusting the parallelism of those tasks. The parallelism of the tasks refers to the maximum number of tasks executed in parallel or the maximum amount of data that the tasks process in parallel; the amount of interactive data generated by the tasks can be controlled by adjusting the parallelism of the tasks.

[0085] Specifically, when a target computing node includes an execution task, the parallelism of the execution task includes the maximum amount of data that the task can process in parallel. This maximum amount of data is determined based on the specific execution information of the task. For example, if the execution information is the resource utilization of the task, such as CPU utilization, a mapping relationship is established between the resource utilization and the maximum amount of data that can be processed in parallel. After determining the execution information, this mapping relationship is used to determine the associated maximum amount of data, and the parallelism of the execution task in the target computing node is adjusted according to this maximum amount of data. The mapping relationship can be determined based on the actual application scenario and usage, and is not limited here.

[0086] When the target computing node includes at least two execution tasks, the parallelism of the execution tasks includes the maximum number of tasks that can be executed in parallel and / or the maximum amount of data that can be processed in parallel by the execution tasks. The strategy for adjusting the parallelism is determined based on the specific execution information. For example, when the output buffer resources are insufficient, one of the execution tasks can be paused, or the maximum amount of data that one of the execution tasks can process in parallel can be reduced.

[0087] In another optional implementation of this embodiment, S530 includes:

[0088] Determine the target tasks in the target computing node based on the execution information;

[0089] Reduce the amount of data processed for the target task.

[0090] When adjusting the parallelism of tasks executed on a target computing node by reducing the amount of data processed, it is necessary to identify the task corresponding to the reduction in data processing volume. If the target computing node has only one task, then that task is directly identified as the target task, and its data processing volume is reduced. If the target computing node includes at least two tasks, the target task for which the data processing volume needs to be reduced must first be clearly identified.

[0091] To ensure the accuracy of target task determination, this embodiment uses the execution information of each task to determine the target task. This reduces the amount of data processed by the target task, effectively decreasing the amount of interactive data generated by the target computing node that needs to be sent to other computing nodes. Furthermore, by reducing the amount of data processed by the target task, the impact on engine performance is minimized.

[0092] For example, such as Figure 3 The diagram illustrates the adjustment of input buffers, output buffers, and task concurrency. When the scheduling node detects that the resource utilization of the output buffer in compute node 2 exceeds a second preset resource utilization threshold, it indicates that tasks 2 and 3 in compute node 2 generate a large amount of interactive data that needs to be stored in the output buffer before being sent to compute node 1, leading to insufficient memory in the output buffer of compute node 2. Therefore, the parallelism of task execution in compute node 2 is dynamically adjusted. For example, the scheduling node reduces the amount of data sent to task 3, i.e., reduces the amount of interactive data generated by task 3 stored in the output buffer, to maintain the resource utilization of the output buffer of compute node 2 below the second preset resource utilization threshold. Once the output buffer has sent the interactive data to the input buffer of compute node 1, making the output buffer resource sufficient, the parallelism of task execution in compute node 2 is readjusted. For example, when the resource utilization of the output buffer reaches a fourth preset resource utilization threshold, the data amount sent by the scheduling node to task 3 is restored.

[0093] If the output buffer is full, data processing will stop and valuable memory will be consumed. This disclosure compensates for the performance loss caused by the inability of computing nodes to increase the rate of consumption of data by generating data.

[0094] In another optional implementation of this embodiment, the execution information includes at least one of the following: execution start time, amount of data processed, and resource utilization rate;

[0095] The target task in the target computing node is determined based on the execution information, including at least one of the following:

[0096] Based on the execution start time, sort the tasks currently being executed on the target computing node in descending order, and determine the task at the top of the sorted results as the target task;

[0097] Based on the amount of data being processed, sort the tasks currently being executed on the target computing node in descending order, and determine the task that is first in the sorting results as the target task.

[0098] Based on resource utilization, the tasks currently being executed on the target computing node are sorted in descending order, and the task ranked first in the sorting results is determined as the target task.

[0099] When determining the target task in the target computing node, the determination is based on at least one piece of information in the execution information, such as the execution start time, the amount of data processed, and / or resource utilization.

[0100] Specifically, the target task can be selected from the following: the task with the latest start time on the target computing node; the task processing the most data on the target computing node; or the task with the highest resource utilization on the target computing node. Resource utilization includes memory utilization, network resource utilization, and CPU utilization. When the target task is determined based on at least two of the execution information, it can be determined according to the weights of the execution information. The specific determination strategy can be determined based on the actual situation and is not limited here.

[0101] By determining the target task based on its startup time, data processing volume, and resource utilization, the resource utilization status of the output buffer can be controlled by adjusting the data processing volume of the target task. This reduces the adverse effects of the adjustment strategy on cross-node data interaction, improves the efficiency of cross-node data interaction, and enhances the fairness of network resource sharing.

[0102] The solution in this embodiment monitors the utilization status of the output buffer resources in the computing nodes to monitor the efficiency of cross-node interactive data generated by the computing nodes. Based on the monitoring results, it adjusts the parallelism of tasks executed within the computing nodes and adjusts the sending of interactive data to the output buffer in a timely manner. This avoids the phenomenon of the output buffer occupying resources for a long time due to differences in the data processing capabilities of upstream and downstream nodes, which would cause data processing to stop and occupy additional memory. By directly adjusting the parallelism based on performance indicators, the efficiency of cross-node data exchange is effectively improved, greatly enhancing the usability of the distributed interactive analysis engine and increasing the fairness of network resource sharing.

[0103] Figure 6 This is a schematic diagram of a task scheduling device according to an embodiment of the present disclosure. This device can execute the task scheduling method involved in any embodiment of the present disclosure; see reference. Figure 6 The task scheduling device 600 includes a buffer resource determination module 610 and an execution task adjustment module 620.

[0104] The buffer resource determination module 610 is used to obtain the resource utilization status of the input buffer in the target computing node; wherein, the input buffer is a data storage area pre-divided in the target computing node for storing data interacting with other computing nodes;

[0105] The task adjustment module 620 is used to adjust the parallelism of tasks executed in other computing nodes according to the resource utilization status of the input buffer and a preset scheduling strategy.

[0106] The solution in this embodiment monitors the resource utilization status of the data input buffer in the computing node used to store data for interaction with other computing nodes, thereby monitoring cross-node data interaction data, avoiding the additional overhead caused by cross-node data interaction, and improving the efficiency of cross-node data interaction.

[0107] In an optional implementation of this embodiment, the input buffer is used to store interactive data sent by other computing nodes and received by the target computing node;

[0108] The task adjustment module includes an input adjustment submodule, which specifically includes:

[0109] The upstream computing node determination unit is used to determine the upstream computing node of the target computing node when the resource insufficiency state is identified based on the resource utilization status of the input buffer.

[0110] The input task adjustment unit is used to control the parallelism of the upstream computing node writing data to the target computing node.

[0111] In an optional implementation of this embodiment, the input task adjustment unit is specifically used for:

[0112] The network connection parallelism between the output buffer in the upstream computing node and the input buffer in the target computing node is adjusted; wherein, the output buffer in the upstream computing node is a data storage area pre-defined in the upstream computing node for storing interactive data to be sent from the upstream computing node to other computing nodes; the adjusted network connection parallelism is less than the original network connection parallelism.

[0113] In an optional implementation of this embodiment, the apparatus further includes an output buffer module, comprising:

[0114] The buffer resource determination submodule is used to obtain the resource utilization status of the output buffer in the target computing node; wherein, the output buffer is a data storage area pre-divided in the target computing node for storing data interacting with other computing nodes;

[0115] The task adjustment submodule is used to adjust the parallelism of the tasks executed in the target computing node according to the resource utilization status of the output buffer and a preset scheduling strategy.

[0116] In an optional implementation of this embodiment, the output buffer is used to store the interactive data that the target computing node needs to send to other computing nodes;

[0117] The task adjustment submodule specifically includes:

[0118] The task execution information determination unit is used to determine the execution information of the task being executed in the target computing node when the resource insufficiency status is identified based on the resource utilization status of the output buffer.

[0119] The output task adjustment unit is used to adjust the parallelism of the tasks executed in the target computing node according to the execution information.

[0120] In an optional implementation of this embodiment, the output task adjustment unit specifically includes:

[0121] The target task determination subunit is used to determine the target task in the target computing node based on the execution information.

[0122] The data volume reduction subunit is used to reduce the amount of data processed by the target task.

[0123] In an optional implementation of this embodiment, the execution information includes at least one of the following: execution start time, amount of data processed, and resource utilization rate;

[0124] The target task defines a sub-unit, which executes at least one of the following:

[0125] Based on the execution start time value, sort the tasks being executed in the target computing node in descending order, and determine the task at the top of the sorting results as the target task;

[0126] Based on the amount of data processed, the tasks being executed in the target computing node are sorted in descending order, and the task at the top of the sorting result is determined as the target task.

[0127] Based on the resource utilization rate, the tasks being executed on the target computing node are sorted in descending order, and the task ranked first in the sorting result is determined as the target task.

[0128] In an optional implementation of this embodiment, the resource utilization status includes resource utilization rate, and the resource insufficiency status means that the resource utilization rate of the input buffer is greater than a first preset resource utilization threshold, or the resource utilization rate of the output buffer is greater than a second preset resource utilization threshold.

[0129] In an optional implementation of this embodiment, the execution entity of the device is a scheduling node in a distributed interactive analysis engine. The distributed interactive analysis engine includes a scheduling node and at least two computing nodes, which are used to execute tasks in the distributed interactive analysis engine in parallel.

[0130] The task scheduling device described above can execute the task scheduling method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the task scheduling method provided in any embodiment of this disclosure.

[0131] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0132] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0133] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0134] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0135] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as task scheduling methods. For example, in some embodiments, the task scheduling method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the task scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform method task scheduling by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0142] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0143] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A task scheduling method, comprising: obtaining a resource utilization state of an input buffer in a target computing node; wherein the input buffer is a data storage area pre-divided in the target computing node, and is used to store cross-node interaction data sent by an upstream computing node to the target computing node; the target computing node and the upstream computing node are in a distributed interactive analysis engine; adjusting parallelism of a task executed in the upstream computing node according to the resource utilization state of the input buffer and according to a preset scheduling strategy; the method further comprises: obtaining a resource utilization state of an output buffer in the target computing node; wherein the output buffer is a data storage area pre-divided in the target computing node, and is used to store interaction data to be sent by the target computing node to other computing nodes; adjusting parallelism of a task executed in the target computing node according to the resource utilization state of the output buffer and according to a preset scheduling strategy; wherein the parallelism of the executed task includes a maximum number of tasks being executed in parallel and / or a maximum data amount processed in parallel.

2. The method of claim 1, wherein, the input buffer is used to store interaction data sent by other computing nodes and received by the target computing node; adjusting parallelism of a task executed in other computing nodes according to the resource utilization state of the input buffer and according to a preset scheduling strategy, comprising: when it is identified from the resource utilization state of the input buffer that there is a resource shortage, determining an upstream computing node of the target computing node; controlling parallelism of data written by the upstream computing node to the target computing node.

3. The method of claim 2, wherein, controlling parallelism of data written by the upstream computing node to the target computing node, comprising: adjusting network connection parallelism of network transmission between an output buffer in the upstream computing node and an input buffer in the target computing node; wherein the output buffer in the upstream computing node is a data storage area pre-divided in the upstream computing node, and is used to store interaction data to be sent by the upstream computing node to other computing nodes; the adjusted network connection parallelism is less than the network connection parallelism before adjustment.

4. The method of claim 1, wherein, the output buffer is used to store interaction data to be sent by the target computing node to other computing nodes; adjusting parallelism of a task executed in the target computing node according to the resource utilization state of the output buffer and according to a preset scheduling strategy, comprising: when it is identified from the resource utilization state of the output buffer that there is a resource shortage, determining execution information of a task being executed in the target computing node; adjusting parallelism of a task executed in the target computing node according to the execution information.

5. The method of claim 4, wherein, adjusting parallelism of a task executed in the target computing node according to the execution information, comprising: determining a target task in the target computing node according to the execution information; reducing a data amount processed by the target task.

6. The method of claim 5, wherein, the execution information includes at least one of the following: execution start time, data amount processed, and resource usage rate; determining a target task in the target computing node according to the execution information, comprising at least one of the following: According to the numerical value of the execution start time, tasks being executed in the target computing node are sorted in descending order, and a task located at the first position in the sorting result is determined as a target task; According to the processing data amount, tasks being executed in the target computing node are sorted in descending order, and a task located at the first position in the sorting result is determined as a target task; According to the resource usage rate, tasks being executed in the target computing node are sorted in descending order, and a task located at the first position in the sorting result is determined as a target task.

7. The method of claim 1 or 4, wherein, The resource utilization state includes a resource utilization rate, and the resource shortage state refers to that the resource utilization rate of the input buffer is greater than a first preset resource utilization threshold, or the resource utilization rate of the output buffer is greater than a second preset resource utilization threshold.

8. The method of claim 1, wherein, The execution subject of the method is a scheduling node in a distributed interactive analysis engine, the distributed interactive analysis engine includes the scheduling node and at least two computing nodes, and the computing nodes are used for executing tasks in the distributed interactive analysis engine in parallel.

9. A task scheduling apparatus, comprising: a buffer resource determination module configured to acquire a resource utilization state of an input buffer in a target computing node; wherein the input buffer is a data storage area pre-divided in the target computing node, and is used for storing cross-node interaction data sent by an upstream computing node to the target computing node; the target computing node and the upstream computing node are in a distributed interactive analysis engine; an execution task adjustment module configured to adjust parallelism of an executed task in the upstream computing node according to the resource utilization state of the input buffer and according to a preset scheduling policy; The apparatus further comprises an output buffer module, comprising: a buffer resource determination submodule configured to acquire a resource utilization state of an output buffer in a target computing node; wherein the output buffer is a data storage area pre-divided in the target computing node, and is used for storing interaction data to be sent by the target computing node to other computing nodes; an execution task adjustment submodule configured to adjust parallelism of an executed task in the target computing node according to the resource utilization state of the output buffer and according to a preset scheduling policy; wherein the parallelism of the executed task includes a maximum number of tasks being executed in parallel and / or a maximum data amount processed in parallel by the executed task.

10. The apparatus of claim 9, wherein, The input buffer is used for storing interaction data sent by other computing nodes and received by the target computing node; The execution task adjustment module comprises an input adjustment submodule, specifically comprising: an upstream computing node determination unit configured to, when a resource shortage state is identified according to the resource utilization state of the input buffer, determine an upstream computing node of the target computing node; an input task adjustment unit configured to control parallelism of data written by the upstream computing node to the target computing node.

11. The apparatus of claim 10, wherein, The input task adjustment unit is specifically configured to: adjusting network connection parallelism of network transmission between the output buffer in the upstream computing node and the input buffer in the target computing node; wherein the output buffer in the upstream computing node is a data storage area pre-divided in the upstream computing node, used for storing interactive data to be sent by the upstream computing node to other computing nodes; and the adjusted network connection parallelism is less than the network connection parallelism before adjustment.

12. The apparatus of claim 9, wherein, The output buffer is used for storing interactive data to be sent by the target computing node to other computing nodes. The task execution adjustment submodule specifically comprises: a task execution information determination unit configured to determine execution information of a task being executed in the target computing node when it is identified that the resource utilization state of the output buffer is in a resource insufficient state; an output task adjustment unit configured to adjust parallelism of the task being executed in the target computing node according to the execution information.

13. The apparatus of claim 12, wherein, The output task adjustment unit specifically comprises: a target task determination subunit configured to determine a target task in the target computing node according to the execution information; a data volume reduction subunit configured to reduce processing data volume of the target task.

14. The apparatus of claim 13, wherein, The execution information comprises at least one of the following: execution start time, processing data volume and resource usage rate. The target task determination subunit performs at least one of the following: sorts tasks being executed in the target computing node in descending order according to the value of the execution start time, and determines a task located at the first place in the sorting result as the target task; sorts tasks being executed in the target computing node in descending order according to the processing data volume, and determines a task located at the first place in the sorting result as the target task; sorts tasks being executed in the target computing node in descending order according to the resource usage rate, and determines a task located at the first place in the sorting result as the target task.

15. The apparatus of claim 10 or 12, wherein, The resource utilization state comprises resource utilization rate, and the resource insufficient state refers to that the resource utilization rate of the input buffer is greater than a first preset resource utilization threshold, or the resource utilization rate of the output buffer is greater than a second preset resource utilization threshold.

16. The apparatus of claim 9, wherein, The execution subject of the device is a scheduling node in a distributed interactive analysis engine, the distributed interactive analysis engine comprises the scheduling node and at least two computing nodes, and the computing nodes are used for executing tasks in the distributed interactive analysis engine in parallel. 17.An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method of any one of claims 1-8.

19. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1-8.

Citation Information

Patent Citations

  • Method for dynamic stationary flow processing system and upstream processing node

    CN101089819A

  • Data processing job scheduling method and device, computer equipment and storage medium

    CN110297701A

  • Data synchronization system and method and computer readable storage medium

    CN111797158A