Resource allocation method and apparatus, storage medium, and electronic device

Abstract

Embodiments of the present application provide a resource allocation method and device, a storage medium and an electronic device, relating to the field of computers, comprising: obtaining first task data from a distributed stream processing system, wherein the first task data is used to describe task information of a current round of tasks; inputting the first task data and a target prediction weight into a target prediction model to obtain a first predicted execution time of a next round of tasks and a first predicted weight of resource allocation, wherein the target prediction weight is used to represent task attributes of the current round of tasks and a priority of a resource allocation request required to ensure allocation of resources for executing the current round of tasks; and allocating target resources required in an execution process of the next round of tasks based on the first predicted execution time and the first predicted weight.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and more specifically, to a resource allocation method and apparatus, a storage medium and an electronic device. Background Technology

[0002] In distributed stream processing systems, such as Apache Flink, real-time data processing, streaming data processing, and batch processing are supported. It provides a flexible computing framework and powerful computing capabilities, thereby supporting a large number of real-time and analytical tasks.

[0003] However, real-time and analytical tasks typically exhibit significant fluctuations in resource requirements. For instance, real-time tasks usually take longer and require more resources than analytical tasks, while analytical tasks typically require less and require fewer resources than real-time tasks. Due to the volatility and unpredictability of resource demands, when using manual resource allocation in existing distributed stream processing systems for task scheduling, the speed of resource allocation cannot match the actual execution time intervals between a large number of real-time tasks. This makes it impossible to ensure that each real-time task is allocated appropriate resources, resulting in low accuracy in resource allocation.

[0004] There is currently no effective solution to the problem of low accuracy in resource allocation in distributed stream processing systems. Summary of the Invention

[0005] This application provides a resource allocation method, apparatus, storage medium, and electronic device to at least address the problem of low accuracy in resource allocation during distributed stream processing.

[0006] According to one embodiment of this application, a resource allocation method is provided, comprising: obtaining a first set of task data from a distributed stream processing system, wherein the first set of task data is used to describe task information of the current round of tasks; and obtaining a first prediction execution of the next round of tasks by inputting the first set of task data and target prediction weights into a target prediction model.

[0007] The first prediction weight for execution time and resource allocation, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating resources to ensure the execution of the current round of tasks; based on the first prediction execution time and the first prediction weight, the target resources required for the execution of the next round of tasks are allocated.

[0008] In an exemplary embodiment, obtaining the first set of task data from the distributed stream processing system includes: transmitting the current round of tasks to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system; decomposing the job graph of the current round of tasks into executable task execution graphs based on the job manager; allocating execution resources to the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight, wherein the execution resources are the total resources required to execute each execution graph in the task execution graph; and obtaining the first set of task data during the execution of the current round of tasks based on the execution resources.

[0009] In an exemplary embodiment, the above-mentioned allocation of execution resources for the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight includes: sending a resource request to the resource manager in the distributed stream processing system based on the target resource configuration parameters, wherein the resource request indicates the desired number of resources indicated by the target resource configuration parameters to be obtained from the resource manager; obtaining the target number of resources from the resource manager based on the resource request, and determining the target number of resources as the execution resources allocated for the current round of tasks.

[0010] In an exemplary embodiment, obtaining the first set of task data from the distributed stream processing system further includes: transmitting the current round of tasks to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system; decomposing the job graph of the current round of tasks into an executable task execution graph based on the job manager; obtaining pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters pre-set during the development of the distributed stream processing system; allocating execution resources for the current round of tasks based on the default resource configuration parameters; and executing the current round of tasks based on the execution resources and generating the first set of task data.

[0011] In an exemplary embodiment, the above-described method of inputting a first set of task data and target prediction weights into a target prediction model to obtain the first predicted execution time and the first predicted weights for resource allocation in the next round of tasks includes: obtaining a second predicted execution time and a second predicted weight output by the target prediction model, wherein the second predicted weight represents the task attributes of the previous round of tasks predicted by the target prediction model and the priority of resource allocation requests when allocating resources to ensure the execution of the previous round of tasks, the previous round of tasks being a round of tasks executed before the current round of tasks; obtaining a second set of task data from a distributed stream processing system, wherein the second set of task data is used to describe the task information of the previous round of tasks; obtaining the target prediction weights and the target predicted execution time by inputting the second set of task data and the second predicted weights into the target prediction model; and obtaining the first predicted execution time and the first predicted weights by inputting the first set of task data and the target predicted weights into the target prediction model.

[0012] In an exemplary embodiment, after obtaining the first prediction execution time and the first prediction weight by inputting the first set of task data and the target prediction weight into the target prediction model, the method further includes: generating a task execution time curve based on the first prediction execution time and the second prediction execution time; determining that the performance of the target prediction model meets a preset requirement when the task execution time curve indicates that N consecutive prediction execution times tend to the target equilibrium point, wherein N is a positive integer greater than or equal to 1; or generating a task execution time curve based on the first prediction execution time and the second prediction execution time; determining the rate of change of the task execution time curve based on the N prediction execution times in the task execution time curve; determining that the performance of the target prediction model meets the preset requirement when the rate of change of the task execution time curve is less than or equal to a preset threshold; or generating a task execution time curve based on the first prediction execution time and the second prediction execution time; determining the rate of change of the task execution time curve based on the N prediction execution times in the task execution time curve; determining that the performance of the target prediction model meets the preset requirement when the task execution time curve indicates that N consecutive task execution times tend to the target equilibrium point and the rate of change is less than or equal to a preset threshold.

[0013] In an exemplary embodiment, after obtaining the first set of task data from the distributed stream processing system, the method further includes: modifying the first set of task data to obtain a modified third set of task data; and obtaining a third prediction for the next round of tasks by inputting the third set of task data and target prediction weights into the target prediction model.

[0014] Execution time and third prediction weight.

[0015] According to another embodiment of the present application, a resource allocation apparatus is also provided, comprising: a first acquisition unit, configured to acquire a first set of task data from a distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; a first processing unit, configured to obtain a first predicted execution time and a first predicted weight of resource allocation for the next round of tasks by inputting the first set of task data and target prediction weights into a target prediction model, wherein the target prediction weights are used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when ensuring the allocation of resources required for the execution of the current round of tasks; and a second processing unit, configured to allocate target resources required for the execution of the next round of tasks based on the first predicted execution time and the first predicted weights.

[0016] According to yet another embodiment of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0017] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] According to yet another embodiment of this application, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0019] Using the embodiments provided in this application, during the execution of the current round of tasks, based on the first set of task data generated during the execution of the current round of tasks and the target prediction weight of the resource configuration output by the target prediction model in the previous round, the target resources that can meet the needs are dynamically allocated in real time. This solves the problem of large errors caused by manually assessing the task volume for resource allocation, ensures the rationality of resource allocation, and achieves the technical effect of improving the accuracy of resource allocation during the execution of tasks using a distributed stream processing system. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a hardware structure block diagram of a server device according to an embodiment of the resource allocation method of this application; Figure 2 This is a flowchart of an optional resource allocation method according to an embodiment of this application; Figure 3 This is an overall schematic diagram of an optional resource allocation using a target prediction model according to an embodiment of this application; Figure 4 This is a structural block diagram of a resource allocation device according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] The resource allocation method embodiments provided in this application can be executed on server devices or similar computing devices. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a resource allocation method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the resource allocation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0026] To better understand the above resource allocation method, the basic concepts of the technical terms and terminology used in the embodiments of this application will be briefly introduced below.

[0027] Flink: Apache Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipelined runtime system can execute both batch and stream processing programs. Furthermore, Flink's runtime itself also supports the execution of iterative algorithms. JobManager: The JobManager is responsible for receiving jobs submitted by clients and managing scheduling. Typically, only one JobManager exists, provided high availability is not a concern. TaskManager: The task manager is responsible for executing tasks distributed by the task manager and processing data. It is a worker node, so there can be one or more, but usually there are multiple. TaskSlot: A task slot can be understood as the number of CPU cores. It represents a fixed-size subset of computing resources that TaskManager has. When ResourceManager allocates resources, the smallest unit is a slot. Each slot contains a set of CPU and memory resources used by the machine to perform computations. Each task needs to be assigned to a specific slot. ResourceManager: The resource manager is mainly responsible for resource allocation and management. There is only one in a Flink cluster. The resources it manages are the task slots of the TaskManager, which are the TaskSlots mentioned above. JobMaster: The JobMaster is the core component of the JobManager, responsible for handling individual jobs. Multiple jobs can run on the same Flink cluster, and each job has its own JobMaster.

[0028] To address the aforementioned problems in related technologies, this embodiment provides a resource allocation method. The execution entity of this method includes, but is not limited to, a distributed stream processing system. This distributed stream processing system includes, but is not limited to, a Flink program, a job manager, and a task manager. The job manager and task manager are two core components that collaborate to ensure the smooth execution of jobs. The job manager is responsible for job submission, initialization, scheduling, monitoring, and fault recovery, while the task manager is responsible for executing specific tasks.

[0029] like Figure 2 As shown, the process of the resource allocation method proposed in this embodiment includes the following steps S202-S206: Step S202: Obtain the first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; Step S204: By inputting the first set of task data and target prediction weights into the target prediction model, the first predicted execution time and the first predicted weight of resource allocation for the next round of tasks are obtained. The target prediction weights are used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating resources to ensure the execution of the current round of tasks. Among them, the task attributes of the current round of tasks include, but are not limited to, the following: (1) Intensive tasks may require more CPU resources and therefore will be given higher weight; (2) Tasks that process a large amount of data may require more memory and storage resources and therefore may be given higher weight; (3) Some tasks may be more important than other tasks and therefore may be given higher weight to ensure that they can get resources first.

[0030] Step S206: Based on the first predicted execution time and the first predicted weight, allocate the target resources required for the execution process in the next round of tasks.

[0031] Before explaining the above resource allocation methods, let's first... Figure 3 The OSD read / write business process functional modules shown are briefly introduced.

[0032] like Figure 3 As shown, in the technical solution of this application, a target prediction model (which can also be understood as a monitoring training model) is added to the traditional Flink open-source stream processing framework. The resource requirements of subsequent tasks are predicted by using the collected monitoring data (task data generated during task execution). Before the job manager schedules the tasks, the number of slots requested from the resource manager is dynamically adjusted based on the prediction results output by the target prediction model.

[0033] Specifically, it includes the following steps: S302, the client submits a job to the JobManager through the REST interface provided by the dispatcher; S304, the job manager distributes tasks (which can also be understood as jobs) to the task manager. The specific process by which the Job Manager distributes tasks to the Task Manager is as follows: (1) Job reception and parsing: The job manager receives job requests submitted by the client and parses the job graph. (2) Job planning: The job manager converts the job graph into an execution graph and determines which task managers each task should be executed on, and how many task slots should be allocated to each task manager, based on job requirements and cluster resource status. (3) Task scheduling: The job manager assigns the tasks in the execution graph to each task manager, starts task execution, and continuously tracks the task status; (4) Resource allocation: The job manager determines resource allocation based on the cluster resource status and job requirements, including determining the number of task slots on each task manager; (5) Execution and monitoring: After receiving a task, the task manager starts executing the task and maintains communication with the job manager during the execution process, reporting the task status and progress.

[0034] S306, execute the current round of tasks through the Task Manager and obtain the first set of task data; When executing tasks, the Task Manager uses task slots to isolate and manage task execution. Each task slot represents a subset of the Task Manager's resources and can independently execute one or more subtasks. The number of task slots determines the number of tasks the Task Manager can execute in parallel.

[0035] This collaborative mechanism between the job manager and the task manager ensures that Flink jobs can run efficiently and reliably in a distributed environment, while providing good fault tolerance and high availability.

[0036] S308, after the task is completed, the Task Manager reports the execution time and various data generated during the execution process to the Job Manager; S310 collects the task data generated during the execution of the current round of tasks by the task manager, and transmits the first set of collected task data to the target prediction model; Before executing the current round of tasks using the task manager, the previous round of tasks was also executed using the task manager. The task data generated during the execution of the previous round of tasks was then input into the target prediction model to obtain the predicted execution time and target prediction weights of the resource configuration for the current round of tasks.

[0037] Target prediction weights are typically used to measure the resource requirements and importance of a task. These prediction weights can help the resource scheduler (which can also be understood as a resource manager) decide how to allocate limited resources to ensure that critical tasks can obtain enough resources to meet their needs, while also maximizing the overall utilization of resources.

[0038] In the process of generating prediction weights using a target prediction model, factors may be considered, but are not limited to, the following: 1) Computational complexity of the task: Computationally intensive tasks may require more CPU resources and therefore may be given higher weight; 2) Data throughput requirements: Tasks that require processing large amounts of data may require more memory and storage resources, and therefore may be given higher weight. 3) Quality of Service (QoS) requirements: Latency-sensitive tasks may require priority resource allocation to ensure they meet QoS requirements; 4) Task Priority: Some tasks may be more important than others, so they may be given higher weight to ensure that they can get resources first.

[0039] The resource manager dynamically allocates resources based on predicted weights and the actual needs of the tasks. This process typically involves complex algorithms and strategies to achieve optimal resource allocation. For example, in Flink, the ResourceManager is responsible for providing, reclaiming, and allocating resources. It manages task slots, which are the unit of resource scheduling in Flink. By adjusting the number of task slots, users can define how subtasks are isolated from each other, thereby affecting resource allocation and task execution.

[0040] In practical distributed systems, resource scheduling is a dynamic and complex process that requires consideration of various factors, including task characteristics, resource availability, and system load. Effective resource scheduling can improve the performance and stability of distributed stream processing systems, as well as increase resource utilization and cost-effectiveness.

[0041] S312, feeds back the first prediction weight output by the target prediction model to the job manager, and allocates the target resources required for the execution process for the next round of tasks based on the first prediction weight.

[0042] The first prediction weight is the prediction weight of resource allocation for the next round of tasks, obtained by inputting the first set of task data and the target prediction weight into the target prediction model.

[0043] S314, through the job manager, feeds back the execution results or calculation results of each round of tasks to the Flink program.

[0044] By adopting the above method, during the execution of the current round of tasks, based on the first set of task data generated during the execution of the current round of tasks and the target prediction weights of resource allocation output by the target prediction model in the previous round, the target resources that can meet the needs are dynamically allocated in real time. This solves the problem of large errors caused by manually assessing the task volume for resource allocation, ensures the rationality of resource allocation, and achieves the technical effect of improving the accuracy of resource allocation during the execution of tasks using a distributed stream processing system.

[0045] In one example embodiment, obtaining the first set of task data from the distributed stream processing system includes: The current round of tasks is transmitted to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system. Based on the job manager, the job graph of the current round of tasks is decomposed into an executable task execution graph; Based on the target resource allocation parameters indicated by the target prediction weight, execution resources are allocated to the current round of tasks. The execution resources are the total resources required for each execution graph in the task execution graph. During the execution of the current round of tasks based on execution resources, the first set of task data is obtained.

[0046] like Figure 3 As shown, the client (APP or WEB) submits jobs or tasks to the job manager through the REST interface provided by the dispatcher. The REST interface is the interface provided by the dispatcher in Flink, which is used to submit jobs and manage running jobs (or tasks).

[0047] This REST interface allows users to interact with the Flink cluster via HTTP requests, including submitting new jobs, canceling jobs, and triggering checkpoints. The dispatcher, acting as an HTTP access point to the cluster, can run across jobs. Upon receiving a job submission request, it starts the job and hands it over to a JobManager for processing.

[0048] The dispatcher's REST API provides a variety of functionalities, such as retrieving job configurations, logs, status, and the savepoint for triggering jobs. These APIs make interacting with the Flink cluster more flexible and convenient, and also support programmatic interaction with the Flink cluster.

[0049] For detailed information on the REST APIs and available operations, please refer to Flink's official documentation, which provides comprehensive API descriptions and usage examples. These REST APIs allow users to effectively manage and monitor the execution of Flink jobs.

[0050] The job manager is launched via the dispatcher, and the job (including the JobGraph) is submitted to the JobMaster. The JobMaster first decomposes the JobGraph into executable graphs (execution graphs or task execution graphs), and then queries and receives the prediction weights from the output of the target prediction model. If the target prediction weights exist, the corresponding execution resources are requested from the resource manager according to the target resource configuration parameters indicated by the target prediction weights; if the target prediction weights are not found, the corresponding default execution resources are requested from the resource manager according to the default resource configuration parameters. The implementation process of requesting execution resources according to the default resource configuration parameters will be described in detail below with reference to specific embodiments.

[0051] It should be noted that during the execution of a task using the execution resources allocated in the current round of tasks, a monitoring task will be initiated. This monitoring task will collect some task data (or monitoring metrics) during the task execution process, such as operator processing speed, task running time, checkpoint health, number of task restarts, JVM and container metrics, etc. The collected task data will be imported into the target prediction model, and the model will be trained to output prediction metrics and recommended parameters (i.e., prediction weights and prediction execution time).

[0052] The collaboration mechanism between the job manager and task manager described above ensures that Flink jobs can run efficiently and reliably in a distributed environment, while providing good fault tolerance and high availability.

[0053] In one example embodiment, the above-mentioned target resource configuration parameters based on the target prediction weight indication allocate execution resources for the current round of tasks, including: Based on the target resource configuration parameters, a resource request is sent to the resource manager in the distributed stream processing system, where the resource request indicates the amount of resources expected to be obtained from the resource manager as indicated by the target resource configuration parameters; Based on the resource request, obtain the target number of resources from the resource manager and determine the target number of resources as the execution resources allocated for the current round of tasks.

[0054] As described in the above embodiments, after the job manager is started via the dispatcher and the job (including the JobGraph) is submitted to the JobMaster, the JobMaster first decomposes the JobGraph into executable Graphs (execution graphs or task execution graphs), and then queries and receives the prediction weights from the output of the target prediction model. If the target prediction weights are found, the corresponding execution resources are requested from the resource manager according to the target resource configuration parameters indicated by the target prediction weights.

[0055] After Task Manager starts, it registers its available slots with Resource Manager. Resource Manager notifies Task Manager of the executable resources required to execute a task. Task Manager connects to Job Manager and provides its own executable resources. Job Manager distributes the tasks to be executed to Task Manager, and Task Manager executes the tasks.

[0056] Clearly, the Task Manager and Job Manager can interact with each other and start a monitoring task for each Task Manager to collect task data or monitoring metrics generated during task execution, such as operator processing speed, task running time, number of task restarts, etc., and import the data into the target prediction model to train the model.

[0057] It should be noted that in practical applications, one task manager may be used to execute one round of tasks, or two or more task managers connected in series may be used to execute the same round of tasks. This application does not limit this.

[0058] In this embodiment, by utilizing the target resource configuration parameters indicated by the target prediction weight output by the target prediction model, the resources required to execute the next round of tasks are evaluated, and a preset number of task slots are automatically configured based on the evaluation results, thus avoiding business anomalies caused by idle resources or task scheduling configurations exceeding the existing resource capacity.

[0059] In addition, compared with the existing technology that manually configures the number of task slots and job scheduling mode, the resource utilization rate cannot be fully utilized due to inaccurate estimation of resource consumption caused by manual assessment of the workload, resulting in resource idleness, the resource allocation method based on the prediction weights dynamically output in real time by the target prediction model in this embodiment is more reasonable and improves the accuracy of resource allocation results.

[0060] In one example embodiment, the above-described method of obtaining the first set of task data from the distributed stream processing system further includes: The current round of tasks is transmitted to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system. Based on the job manager, the job graph of the current round of tasks is decomposed into an executable task execution graph; Retrieve pre-configured default resource configuration parameters, which are parameters that are pre-set during the development of the distributed stream processing system; Based on the default resource configuration parameters, allocate execution resources for the current round of tasks; Based on the execution resources, execute the current round of tasks and generate the first set of task data.

[0061] As described in the above embodiments, the job manager is started by the dispatcher, and the job (including the JobGraph) is submitted to the JobMaster. The JobMaster first decomposes the JobGraph into executable graphs (execution graphs or task execution graphs), and then queries and receives the prediction weights from the output of the target prediction model. If the target prediction weights are not found, the corresponding default execution resources are requested from the resource manager according to the default resource configuration parameters, and execution resources are allocated for the current round of tasks according to the default resource configuration parameters, thereby executing the current round of tasks.

[0062] The specific steps for executing a Flink task using the default resource configuration parameters are as follows: S11, the client (APP or WEB) submits the job to the JobManager through the REST interface provided by the dispatcher; S12, the dispatcher starts the JobMaster and submits the job (including the JobGraph) to the JobMaster; S13, JobMaster first breaks down JobGraph into executable Graphs and requests the corresponding execution resources (slots) from the resource manager. S14, Determine if the current resource manager has enough resources. If not, start a new TaskManager; S15, After TaskManager starts, it registers its available slots with the resource manager; S16, the resource manager notifies the task manager that the job should provide executable resources; S17, TaskManager connects to JobMaster and provides its own executable resources; S18, TaskManager connects to JobMaster and provides its own executable resources; S19, TaskManager executes tasks and can exchange data with each other; S20 enables the monitoring of certain metrics and provides a UI for viewing them.

[0063] By using pre-configured default resource parameters, it can be ensured that execution resources are still allocated to the current round of tasks according to the default configuration even when the monitoring task is not started. This guarantees the execution process of the tasks, ensures that the distributed stream processing system can adapt to constantly changing workloads and environmental conditions, and improves the stability of the system.

[0064] In one example embodiment, the above method of inputting the first set of task data and target prediction weights into the target prediction model to obtain the first predicted execution time and the first predicted weights for resource allocation in the next round of tasks includes: Obtain the second prediction execution time and the second prediction weight output by the target prediction model. The second prediction weight represents the task attributes of the previous round of tasks predicted by the target prediction model and the priority of the resource allocation request when allocating the resources required to ensure the execution of the previous round of tasks. The previous round of tasks is the round of tasks executed before the current round of tasks. Obtain the second set of task data from the distributed stream processing system, where the second set of task data is used to describe the task information of the previous round of tasks; By inputting the second set of task data and the second prediction weights into the target prediction model, the target prediction weights and target prediction execution time are obtained. By inputting the first set of task data and target prediction weights into the target prediction model, the first prediction execution time and the first prediction weights are obtained.

[0065] In this application embodiment, the H20.AI machine learning model platform can be used as the target prediction model, but is not limited to. The model platform provides an interface for integration with Flink. By designing the model and using gradient boosting learning technology, the execution time of Flink tasks is imported to regress and predict the execution time of future tasks. The failure count and failure nodes of Flink task checkpoints are imported through decision tree learning technology to judge and predict the weight of resource allocation for future tasks.

[0066] Obviously, the input to the target prediction model includes, but is not limited to, two parts: the first part is the task data of the current round of tasks being executed, and the second part is the target prediction weight of the current round of tasks output by the target prediction model based on the task data collected during the execution of the previous round of tasks. The output of the target prediction model includes the first prediction execution time and the first prediction weight of the next round of tasks.

[0067] In other words, the methods for obtaining the target prediction weight of the current round of tasks include: obtaining the task data of the previous round of tasks, and obtaining the second prediction execution time and the second prediction weight output by the target prediction model; by inputting the second set of task data and the second prediction weight into the target prediction model, the target prediction weight and target prediction execution time of the current round of tasks are obtained.

[0068] After obtaining the target prediction weights output by the target prediction model, the first set of task data and target prediction weights are input into the target prediction model to obtain the first prediction execution time and the first prediction weights output by the model.

[0069] In other words, by using the prediction weights output by the target prediction model in the previous training round and the task data of the current round as inputs to the model in the current training round, the target prediction model is continuously trained and optimized. This allows the monitoring data after each task execution to be fed back to the target prediction model for further training and optimization, thereby improving the model's prediction accuracy and stability.

[0070] Meanwhile, by collecting system monitoring data in real time and using machine learning models for prediction, precise adjustments to dynamic resource requests are achieved, effectively improving the resource utilization and job processing efficiency of the distributed stream processing system. Actual results show that after adopting the technical solution in this embodiment, system resource waste is significantly reduced, job processing latency is significantly lowered, and overall performance is significantly improved. Furthermore, the continuous training and optimization mechanism of the model ensures that the system can adapt to constantly changing workloads and environmental conditions, improving system stability and flexibility, and providing strong support for large-scale data processing.

[0071] In one example embodiment, after obtaining the first prediction execution time and the first prediction weights by inputting the first set of task data and target prediction weights into the target prediction model, the above method further includes: Based on the first and second predicted execution times, a task execution time curve is generated. If the task execution time curve represents N consecutive prediction execution times that all tend towards the target equilibrium point, then the performance of the target prediction model is determined to meet the preset requirements, where N is a positive integer greater than or equal to 1; or Based on the first and second predicted execution times, a task execution time curve is generated. Based on the N predicted execution times in the task execution time curve, determine the rate of change of the task execution time curve. If the rate of change of the task execution time curve is less than or equal to a preset threshold, the performance of the target prediction model is determined to meet the preset requirements; or Based on the first and second predicted execution times, a task execution time curve is generated. Based on the N predicted execution times in the task execution time curve, determine the rate of change of the task execution time curve. If the task execution time curve indicates that the execution times of N consecutive predictions all tend to the target equilibrium point and the rate of change is less than or equal to the preset threshold, then the performance of the target prediction model is determined to meet the preset requirements.

[0072] As described in the above embodiments, after each task execution, the generated task data is used to train the target prediction model in real time. So how to evaluate the accuracy of the model prediction, and how to ensure that the model can be adaptively adjusted for different business scenarios? The embodiments of this application provide the following possible evaluation methods.

[0073] For example, after each task execution, in addition to generating the prediction weights for the resource allocation of the next round of tasks, the target prediction model also generates the prediction execution time of the next round of tasks, records the prediction execution time output in each training process, and then evaluates whether the training results of the model tend to be more accurate in the next output resource allocation results based on a set of prediction execution times.

[0074] Specifically, when the execution times of N consecutive predictions all tend to the target equilibrium point, it can be determined that the training process of the target prediction model tends to continuously optimize the direction of outputting better resource allocation results in the next iteration, and the performance of the target prediction model can meet the preset requirements.

[0075] For example, by using the task execution time curve generated from the recorded N predicted execution times, the rate of change of the task time curve can be determined. The rate of change of the time curve typically refers to the volatility or stability of the time series data. If the rate of change of the time curve is small, it usually means that the changes between data points are not significant, or that the data volatility is low. This can be interpreted as good stability of the time series, or that the time series is more stable.

[0076] However, it should be noted that a small rate of change does not necessarily mean that the time series is completely stationary. Therefore, this embodiment also proposes another method to evaluate the model training results. That is, when the execution times of N consecutive tasks represented by the task execution time curve all tend to the target equilibrium point and the rate of change of the task execution time curve is less than or equal to a preset threshold, it is determined that the performance of the target prediction model meets the preset requirements.

[0077] Therefore, by using the above evaluation methods, and ensuring that the training process of the target prediction model tends to produce better resource allocation results in the next iteration, a more optimized target prediction model is used to predict the resource requirements of subsequent tasks. Specifically, before task scheduling, the number of slots requested from the resource manager is dynamically adjusted based on the prediction results of the target prediction model, thereby dynamically adjusting the execution resources for the next round of task allocation and improving the accuracy of resource allocation.

[0078] Furthermore, after each task execution, the collected monitoring data (task data) is fed back to the target prediction model, which can continuously train and optimize the prediction capability of the target prediction model. This continuous training and optimization mechanism ensures that the distributed stream processing system can adapt to constantly changing workloads and environmental conditions, thereby improving the stability of the system.

[0079] In another alternative embodiment, after obtaining the first set of task data from the distributed stream processing system, the method further includes: Modify the first set of task data to obtain the modified third set of task data; By inputting the third set of task data and target prediction weights into the target prediction model, the third prediction execution time and third prediction weights for the next round of tasks are obtained.

[0080] To ensure that different training can be conducted for different business needs, in practical applications, the data collected by the current target prediction model can be added or removed for different business needs. By using the task data after addition or removal, the input data of the model can be flexibly determined, thereby adjusting the training direction of the model. This ensures that the final prediction results and the prediction weights of resource allocation include the computing and storage resources used by each node.

[0081] Furthermore, it allows for the most rational allocation of physical resources in the production environment, enhancing the flexibility of the target prediction model and enabling it to adapt to different business scenarios, thus ensuring the smooth execution of tasks.

[0082] As described in the above embodiments, the technical solution of this application uses existing task data generated during task execution and inputs it into the target prediction model, enabling the target prediction model to provide recommended scheduling strategies and resource allocation. In other words, by reusing the collected task data, the process can be optimized based on existing data without increasing resource consumption.

[0083] Furthermore, based on the task data collected during task execution, the model trained in each round is more accurate and efficient in resource allocation than task experience suggests, reducing manpower costs and achieving automated task allocation. Moreover, the performance and accuracy of the target prediction model improve synchronously with the number of task executions. Therefore, the scheduling efficiency and accuracy of the task using the improved distributed stream processing system in this embodiment will continue to increase positively.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0085] This embodiment also provides a resource allocation device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the modules described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0086] Figure 4 This is a structural block diagram of a resource allocation device according to an embodiment of this application. The device includes: The first acquisition unit 402 is used to acquire a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; The first processing unit 404 is used to input the first set of task data and target prediction weights into the target prediction model to obtain the first predicted execution time and the first predicted weight of resource allocation for the next round of tasks. The target prediction weights are used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating resources to ensure the execution of the current round of tasks. The second processing unit 406 is used to allocate the target resources required for the execution process of the next round of tasks based on the first predicted execution time and the first predicted weight.

[0087] In an exemplary embodiment, the first acquisition unit 402 includes: a first transmission module, configured to transmit the current round of task data through the presentation layer state transition interface provided by the distributor of the distributed stream processing system.

[0088] The task is transmitted to the job manager; the first decomposition module is used to decompose the job graph of the current round of tasks into executable task execution graphs based on the job manager; the first allocation module is used to allocate execution resources to the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight, wherein the execution resources are the total resources required when executing each execution graph in the task execution graph; the first acquisition module is used to acquire the first set of task data during the execution of the current round of tasks based on the execution resources.

[0089] In an exemplary embodiment, the first allocation module includes: a sending submodule, configured to send a resource request to a resource manager in a distributed stream processing system based on target resource configuration parameters, wherein the resource request indicates an expectation to obtain the number of resources indicated by the target resource configuration parameters from the resource manager; and a first acquisition submodule, configured to acquire the target number of resources from the resource manager based on the resource request, and determine the target number of resources as the execution resources for the current round of task allocation.

[0090] In an exemplary embodiment, the first acquisition unit 402 includes: a second transmission module, configured to transmit the current round of tasks to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system; a second decomposition module, configured to decompose the job graph of the current round of tasks into an executable task execution graph based on the job manager; a second acquisition module, configured to acquire pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters pre-set during the development of the distributed stream processing system; a second allocation module, configured to allocate execution resources to the current round of tasks based on the default resource configuration parameters; and a first processing module, configured to execute the current round of tasks based on the execution resources and generate a first set of task data.

[0091] In an exemplary embodiment, the first processing unit 404 includes: a third acquisition module, configured to acquire a second prediction execution time and a second prediction weight output by a target prediction model, wherein the second prediction weight represents the task attributes of the previous round of tasks predicted by the target prediction model and the priority of resource allocation requests when allocating resources to ensure the execution of the previous round of tasks, the previous round of tasks being a round of tasks executed before the current round of tasks; a fourth acquisition module, configured to acquire a second set of task data from a distributed stream processing system, wherein the second set of task data is used to describe the task information of the previous round of tasks; by inputting the second set of task data and the second prediction weight into the target prediction model, a target prediction weight and a target prediction execution time are obtained; and a second processing module, configured to input the first set of task data and the target prediction weight into the target prediction model, a first prediction execution time and a first prediction weight are obtained.

[0092] In an exemplary embodiment, the above-described apparatus further includes: a third processing unit, configured to, after obtaining a first prediction execution time and a first prediction weight by inputting a first set of task data and a target prediction weight into a target prediction model, generate a task execution time curve based on the first prediction execution time and a second prediction execution time; determine that the performance of the target prediction model meets a preset requirement when the task execution time curve indicates that N consecutive prediction execution times tend to the target equilibrium point, wherein N is a positive integer greater than or equal to 1; or generate a task execution time curve based on the first prediction execution time and the second prediction execution time; determine the rate of change of the task execution time curve based on the N prediction execution times in the task execution time curve; determine that the performance of the target prediction model meets the preset requirement when the rate of change of the task execution time curve is less than or equal to a preset threshold; or generate a task execution time curve based on the first prediction execution time and the second prediction execution time; determine the rate of change of the task execution time curve based on the N prediction execution times in the task execution time curve; determine that the performance of the target prediction model meets the preset requirement when the task execution time curve indicates that N consecutive task execution times tend to the target equilibrium point and the rate of change is less than or equal to a preset threshold.

[0093] In an exemplary embodiment, the above apparatus further includes: a fourth processing unit, configured to modify the first set of task data after obtaining the first set of task data from the distributed stream processing system to obtain the modified third set of task data; and a fifth processing unit, configured to obtain the third prediction execution time and the third prediction weight of the next round of tasks by inputting the third set of task data and the target prediction weight into the target prediction model.

[0094] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0095] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0096] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps: S1, Obtain the first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; S2, by inputting the first set of task data and target prediction weights into the target prediction model, the first predicted execution time and the first predicted weight of resource allocation for the next round of tasks are obtained. The target prediction weights are used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating resources to ensure the execution of the current round of tasks. S3, based on the first predicted execution time and the first predicted weight, allocates the target resources required for the execution process of the next round of tasks.

[0097] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0098] Embodiments of this application also provide an electronic device, such as... Figure 5 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments via the computer program.

[0099] Optionally, in this embodiment, the processor 504 can be configured to perform the following steps via a computer program: S1, Obtain the first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; S2, by inputting the first set of task data and target prediction weights into the target prediction model, the first predicted execution time and the first predicted weight of resource allocation for the next round of tasks are obtained. The target prediction weights are used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating resources to ensure the execution of the current round of tasks. S3, based on the first predicted execution time and the first predicted weight, allocates the target resources required for the execution process of the next round of tasks.

[0100] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0101] Alternatively, as those skilled in the art will understand, Figure 5 The structure shown is for illustrative purposes only. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 5 The different configurations shown.

[0102] The memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the resource allocation method and resource allocation device in this embodiment. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, thereby implementing the above-mentioned resource allocation method. The memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include memory remotely located relative to the processor 504, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 502 may be used, but is not limited to, to store the first set of task data, target prediction weights, and the first prediction execution time, etc. As an example, such as... Figure 5 As shown, the memory 502 may include, but is not limited to, the first acquisition unit 402, the first processing unit 404, and the second processing unit 406 in the resource allocation device described above. Furthermore, it may include, but is not limited to, other module units in the resource allocation device described above, which will not be elaborated upon in this example.

[0103] Optionally, the transmission device 506 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 506 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 506 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0104] In addition, the above-mentioned electronic device also includes: a display 508; and a connection bus 510 for connecting the various module components in the above-mentioned electronic device.

[0105] In other embodiments, the aforementioned electronic device can be a node in a distributed system, which can be a blockchain system. This blockchain system is formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any type of computing device, such as a server or terminal, can become a node in the blockchain system by joining this peer-to-peer network.

[0106] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0107] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0108] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0109] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0110] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A resource allocation method, characterized in that, include: Obtain the first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; By inputting the first set of task data and target prediction weights into the target prediction model, the first predicted execution time and the first predicted weight of resource allocation for the next round of tasks are obtained. The target prediction weights represent the task attributes of the current round of tasks and the priority of resource allocation requests when ensuring the allocation of resources required for the execution of the current round of tasks. In generating the first prediction weights using the target prediction model, the computational complexity, data throughput requirements, service quality requirements, and task priority of the tasks need to be considered. For computationally intensive tasks, the current round of tasks is assigned a higher weight. Latency-sensitive tasks in the current round of tasks are allocated resources preferentially. When the task priority indicates that some tasks in the current round of tasks are more important than others, those tasks are assigned a higher weight. Based on the first predicted execution time and the first predicted weight, allocate the target resources required for the execution of the next round of tasks; The step of obtaining the first set of task data from the distributed stream processing system includes: transmitting the current round of tasks to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system; decomposing the job graph of the current round of tasks into executable task execution graphs based on the job manager; allocating execution resources to the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight, wherein the execution resources are the total resources required to execute each execution graph in the task execution graph; and obtaining the first set of task data during the execution of the current round of tasks based on the execution resources. The method further includes: during the execution of the current round of tasks using the execution resources, starting a monitoring task, collecting the first set of task data generated during task execution through the monitoring task, and using the first set of task data to train the target prediction model, wherein the first set of task data includes operator processing speed, task running time, checkpoint health, and number of task restarts.

2. The method according to claim 1, characterized in that, The allocation of execution resources for the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight includes: Based on the target resource configuration parameters, a resource request is sent to the resource manager in the distributed stream processing system, wherein the resource request indicates a desire to obtain the quantity of resources indicated by the target resource configuration parameters from the resource manager; Based on the resource request, a target number of resources are obtained from the resource manager, and the target number of resources are determined as the execution resources allocated for the current round of tasks.

3. The method according to claim 1, characterized in that, The step of obtaining the first set of task data from the distributed stream processing system also includes: The current round of tasks is transmitted to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system. Based on the job manager, the job graph of the current round of tasks is decomposed into an executable task execution graph; Obtain pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters that are pre-set during the development of the distributed stream processing system; Based on the default resource configuration parameters, execution resources are allocated to the current round of tasks; Based on the execution resources, the current round of tasks is executed, and the first set of task data is generated.

4. The method according to claim 1, characterized in that, The step of inputting the first set of task data and target prediction weights into the target prediction model to obtain the first predicted execution time and the first predicted weights of resource allocation for the next round of tasks includes: Obtain the second prediction execution time and the second prediction weight output by the target prediction model, wherein the second prediction weight represents the task attributes of the previous round of tasks predicted by the target prediction model and the priority of the resource allocation request when ensuring the allocation of resources required to execute the previous round of tasks, wherein the previous round of tasks is a round of tasks executed before the current round of tasks are executed; Obtain a second set of task data from the distributed stream processing system, wherein the second set of task data is used to describe the task information of the previous round of tasks; By inputting the second set of task data and the second prediction weight into the target prediction model, the target prediction weight and the target prediction execution time are obtained; By inputting the first set of task data and the target prediction weights into the target prediction model, the first prediction execution time and the first prediction weights are obtained.

5. The method according to claim 4, characterized in that, After obtaining the first prediction execution time and the first prediction weight by inputting the first set of task data and the target prediction weight into the target prediction model, the method further includes: Based on the first predicted execution time and the second predicted execution time, a task execution time curve is generated; If the task execution time curve indicates that the execution times of N consecutive predictions all tend towards the target equilibrium point, then the performance of the target prediction model is determined to meet the preset requirements, where N is a positive integer greater than or equal to 1; or The task execution time curve is generated based on the first predicted execution time and the second predicted execution time; Based on the N predicted execution times in the task execution time curve, determine the rate of change of the task execution time curve; If the rate of change of the task execution time curve is less than or equal to a preset threshold, it is determined that the performance of the target prediction model meets the preset requirements; or The task execution time curve is generated based on the first predicted execution time and the second predicted execution time; Based on the N predicted execution times in the task execution time curve, determine the rate of change of the task execution time curve; If the task execution time curve indicates that the execution times of N consecutive tasks tend to the target equilibrium point and the rate of change is less than or equal to the preset threshold, then the performance of the target prediction model is determined to meet the preset requirements.

6. The method according to any one of claims 1 to 5, characterized in that, After obtaining the first set of task data from the distributed stream processing system, the method further includes: Modify the first set of task data to obtain the modified third set of task data; By inputting the third set of task data and the target prediction weights into the target prediction model, the third prediction execution time and the third prediction weights for the next round of tasks are obtained.

7. A resource allocation device, characterized in that, include: The first acquisition unit is used to acquire a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe the task information of the current round of tasks; The first processing unit is configured to input the first set of task data and target prediction weights into a target prediction model to obtain the first predicted execution time and the first predicted weights for resource allocation of the next round of tasks. The target prediction weights represent the task attributes of the current round of tasks and the priority of resource allocation requests when ensuring the allocation of resources required for the execution of the current round of tasks. In generating the first prediction weights using the target prediction model, the computational complexity, data throughput requirements, service quality requirements, and task priority of the tasks need to be considered. For computationally intensive tasks, the current round of tasks is assigned higher weights. Resources are allocated preferentially to latency-sensitive tasks in the current round of tasks. When the task priority indicates that some tasks in the current round of tasks are more important than others, those tasks are assigned higher weights. The second processing unit is configured to allocate target resources required for the execution of the next round of tasks based on the first predicted execution time and the first predicted weight. The first acquisition unit is further configured to transmit the current round of tasks to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system; based on the job manager, decompose the job graph of the current round of tasks into executable task execution graphs; allocate execution resources to the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight, wherein the execution resources are the total resources required to execute each execution graph in the task execution graph; and acquire the first set of task data during the execution of the current round of tasks based on the execution resources. The device is further configured to initiate a monitoring task during the execution of the current round of tasks using the execution resources, collect the first set of task data generated during task execution through the monitoring task, and use the first set of task data to train the target prediction model, wherein the first set of task data includes operator processing speed, task running time, checkpoint health, and number of task restarts.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

Images