Resource allocation method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a resource allocation method and device, a storage medium and electronic equipment, and relates to the field of computers.The method comprises the steps that a first set of task data is obtained from a distributed stream processing system, and the first set of task data is used for describing task information of a current round of task; inputting a first group of task data and a target prediction weight into a target prediction model to obtain first prediction execution time of a next round of task and a first prediction weight of resource configuration, the target prediction weight is used for representing the task attribute of the current round of task and the priority of the resource allocation request when the required resources are allocated for ensuring the execution of the current round of task; and based on the first prediction execution time and the first prediction weight, allocating target resources required in the execution process for the next round of task.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of distributed storage technology, and more specifically, to a resource allocation method and device, a storage medium, and an electronic device. Background Art

[0002] In distributed stream processing systems, for example, Apache Flink supports real-time data processing, stream data processing, and batch processing, and provides a flexible computing framework and powerful computing capabilities, thereby supporting a large number of real-time tasks and analysis tasks.

[0003] However, the resource requirements of real-time tasks and analytical tasks usually fluctuate significantly. For example, the time for real-time tasks to request resources is usually longer than that for analytical tasks, and the resource requirements are larger, while the time for analytical tasks to request resources is usually shorter than that for real-time tasks, and the resource requirements are smaller. Due to the volatility and unpredictability of resource requirements, when manually configuring resource allocation in existing distributed stream processing systems to implement task scheduling, the speed of resource allocation cannot match the actual execution time intervals between a large number of real-time tasks, and it is impossible to ensure that reasonable resources are allocated to each real-time task, resulting in the problem of low accuracy of resource allocation in the resource allocation process.

[0004] In view of the problem of low accuracy of resource allocation in distributed stream processing systems in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] Embodiments of the present application provide a resource allocation method and apparatus, a storage medium, and an electronic device to at least solve the problem of low accuracy of resource allocation in distributed stream processing.

[0006] According to one embodiment of the present application, a resource allocation method is provided, including: obtaining a first set of task data from a distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks; obtaining a first predicted execution time and a first predicted weight of resource configuration for a next round of tasks by inputting the first set of task data and a target prediction weight into a target prediction model, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when ensuring that the resources required for the execution of the current round of tasks are allocated; and allocating target resources required during the execution of the next round of tasks based on the first predicted execution time and the first predicted weight.

[0007] In an exemplary embodiment, the above-mentioned obtaining of a first set of task data from a distributed stream processing system includes: transmitting a current round of tasks to a job manager through a presentation layer state transition interface provided by a distributor of the distributed stream processing system; based on the job manager, decomposing a job graph of the current round of tasks into an executable task execution graph; allocating execution resources to the current round of tasks based on target resource configuration parameters indicated by target prediction weights, wherein the execution resources are the total resources required to execute each execution graph in the task execution graph; in the process of executing the current round of tasks based on the execution resources, obtaining the first set of task data.

[0008] In an exemplary embodiment, the target resource configuration parameters indicated based on the target prediction weights are used to allocate execution resources for the current round of tasks, including: based on the target resource configuration parameters, sending a resource request to a resource manager in a distributed stream processing system, wherein the resource request indicates the expected number of resources indicated by the target resource configuration parameters to be obtained from the resource manager; based on the resource request, obtaining a target number of resources from the resource manager, and determining the target number of resources as the execution resources allocated for the current round of tasks.

[0009] In an exemplary embodiment, the above-mentioned obtaining of the first set of task data from the distributed stream processing system also includes: transmitting the current round of tasks to the job manager through the presentation layer state conversion interface provided by the distributor of the distributed stream processing system; based on the job manager, decomposing the job graph of the current round of tasks into an executable task execution graph; obtaining pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters pre-set during the development of the distributed stream processing system; based on the default resource configuration parameters, allocating execution resources for the current round of tasks; based on the execution resources, executing the current round of tasks and generating the first set of task data.

[0010] In an exemplary embodiment, the above-mentioned method of obtaining the first predicted execution time and the first predicted weight of resource configuration for the next round of tasks by inputting the first set of task data and the target prediction weight into the target prediction model includes: obtaining the second predicted execution time and the second predicted weight output by the target prediction model, wherein the second prediction weight represents the task attributes of the previous round of tasks predicted by the target prediction model and the priority of the resource allocation request when ensuring the allocation of resources required for the execution of the previous round of tasks, and the previous round of tasks is a round of tasks executed before the execution of the current round of tasks; obtaining a second set of task data from the distributed stream processing system, wherein the second set of task data is used to describe the task information of the previous round of tasks; obtaining the target prediction weight and the target predicted execution time by inputting the second set of task data and the second prediction weight into the target prediction model; obtaining the first predicted execution time and the first prediction weight by inputting the first set of task data and the target prediction weight into the target prediction model.

[0011] In an exemplary embodiment, after obtaining the first predicted execution time and the first predicted weight by inputting the first set of task data and the target prediction weight into the target prediction model, the method also includes: generating a task execution time curve based on the first predicted execution time and the second predicted execution time; when the task execution time curve indicates that N consecutive predicted execution times tend to the target balance point, determining that the performance of the target prediction model meets the preset requirements, wherein N is a positive integer greater than or equal to 1; or generating a task execution time curve based on the first predicted execution time and the second predicted execution time; determining the rate of change of the task execution time curve based on the N predicted execution times in the task execution time curve; when the rate of change of the task execution time curve is less than or equal to a preset threshold, determining that the performance of the target prediction model meets the preset requirements; or generating a task execution time curve based on the first predicted execution time and the second predicted execution time; determining the rate of change of the task execution time curve based on the N predicted execution times in the task execution time curve; when the task execution time curve indicates that N consecutive task execution times tend to the target balance point and the rate of change is less than or equal to the preset threshold, determining that the performance of the target prediction model meets the preset requirements.

[0012] In an exemplary embodiment, after obtaining the first set of task data from the distributed stream processing system, the method further includes: modifying the first set of task data to obtain a modified third set of task data; and obtaining a third prediction of the next round of tasks by inputting the third set of task data and the target prediction weight into the target prediction model.

[0013] Execution time and third prediction weight.

[0014] According to another embodiment of the embodiment of the present application, a resource allocation device is also provided, including: a first acquisition unit, used to acquire a first set of task data from a distributed stream processing system, wherein the first set of task data is used to describe the task information of a current round of tasks; a first processing unit, used to obtain a first predicted execution time and a first predicted weight of resource configuration for a next round of tasks by inputting the first set of task data and the target prediction weight into a target prediction model, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests to ensure that the resources required for the execution of the current round of tasks are allocated; a second processing unit, used to allocate the target resources required during the execution of the next round of tasks based on the first predicted execution time and the first prediction weight.

[0015] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0016] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to another embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0018] By adopting the above-mentioned embodiment provided by the present application, in the process of executing the current round of tasks, based on the first set of task data generated in the process of executing the current round of tasks and the target prediction weight of the resource configuration output by the target prediction model in the previous round, the target resources that can meet its needs are dynamically allocated in real time, thereby solving the problem of large errors caused by manual assessment of task volume for resource allocation, ensuring the rationality of resource allocation, and achieving the technical effect of improving the accuracy of resource allocation in the process of executing tasks using a distributed stream processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a hardware structure block diagram of a server device according to a resource allocation method in an embodiment of the present application;

[0021] Figure 2 is a flowchart of an optional resource allocation method according to an embodiment of the present application;

[0022] Figure 3 It is an overall schematic diagram of an optional resource allocation using a target prediction model according to an embodiment of the present application;

[0023] Figure 4 is a structural block diagram of a resource allocation device according to an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The resource allocation method embodiment provided in the embodiment of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a resource allocation method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the resource allocation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In order to better understand the above resource allocation method, the basic concepts of professional terms and terminology used in the embodiments of the present application are briefly introduced below.

[0031] Flink: Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed data flow engine written in Java and Scala. Flink executes any stream data program in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute batch and stream processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms;

[0032] JobManager: Job Manager, which is responsible for receiving jobs submitted by clients and managing scheduling. If high availability is not considered, there is usually only one job manager.

[0033] TaskManager: a task manager that is responsible for executing tasks assigned by the task manager and processing data. It is a working node, so there can be one or more, usually multiple;

[0034] TaskSlot: Task slots can be understood as the number of CPU cores, representing a fixed-size subset of computing resources owned by TaskManager. When ResourceManager allocates resources, the minimum unit is slot. Each slot contains a set of CPU and memory resources used by the machine to perform calculations. Each task needs to be allocated to a specific slot.

[0035] ResourceManger: Resource manager, which is mainly responsible for resource allocation and management. There is only one resource manager in a Flink cluster. The resources it manages are the task slots of TaskManger, which are the TaskSlots mentioned above.

[0036] JobMaster: JobMaster is the core component of JobManager, responsible for processing individual jobs. Multiple jobs can run in the same Flink cluster, and each job has a JobMaster.

[0037] In order to solve the above problems existing in the related art, a resource allocation method is provided in this embodiment, and its execution subject includes but is not limited to a distributed stream processing system, wherein the distributed stream processing system includes but is not limited to a Flink program, a job manager and a task manager, etc. The job manager (JobManager) and the task manager (TaskManager) are two core components, which work together to ensure the smooth execution of the job. The job manager is responsible for the submission, initialization, scheduling, monitoring and fault recovery of the job, while the task manager is responsible for executing specific tasks.

[0038] like Figure 2 As shown, the process of the resource allocation method proposed in this embodiment includes the following steps S202-S206:

[0039] Step S202: obtaining a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks;

[0040] Step S204: obtaining a first predicted execution time and a first predicted weight of resource allocation for the next round of tasks by inputting the first set of task data and the target prediction weight into the target prediction model, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating the resources required for executing the current round of tasks;

[0041] The task attributes of the current round of tasks include but are not limited to the following: (1) Intensive tasks may require more CPU resources and therefore will be given a higher weight; (2) Tasks that process large amounts of data may require more memory and storage resources and therefore may be given a higher weight; (3) Some tasks may be more important than other tasks and therefore may be given a higher weight to ensure that they can obtain resources first.

[0042] Step S206: Allocate target resources required in the execution process for the next round of tasks based on the first predicted execution time and the first predicted weight.

[0043] Before explaining the above resource allocation method, we first Figure 3 The following is a brief introduction to the OSD read and write business process function modules.

[0044] like Figure 3As shown, in the technical solution of the present application, a target prediction model (which can also be understood as a monitoring training model) is added to the traditional Flink open source stream processing framework, and the resource requirements of subsequent tasks are predicted using the collected monitoring data (task data generated during task execution). Before the job manager schedules tasks, the number of slots requested from the resource manager is dynamically adjusted based on the prediction results output by the target prediction model.

[0045] Specifically, the steps include:

[0046] S302, the client submits a job to the job manager (JobManager) through the REST interface provided by the distributor;

[0047] S304, the job manager distributes the task (which can also be understood as a job) to the task manager (TaskManager);

[0048] The specific process of the job manager distributing tasks to the task manager is as follows:

[0049] (1) Job reception and analysis: The job manager receives the job request submitted by the client and parses the job graph (JobGraph);

[0050] (2) Job planning: The job manager converts the job graph into an execution graph and determines which task managers each task should be executed on and how many task slots should be allocated to each task manager based on job requirements and cluster resource conditions.

[0051] (3) Task Scheduling: The job manager assigns tasks in the execution graph to each task manager, starts task execution, and continuously tracks task status;

[0052] (4) Resource allocation: The job manager determines resource allocation based on the cluster resource status and job requirements, including determining the number of task slots on each task manager;

[0053] (5) Execution and monitoring: After receiving the task, the task manager starts to execute the task and maintains communication with the job manager during the execution process to report the task status and progress.

[0054] S306, executing the current round of tasks through the task manager and obtaining a first set of task data;

[0055] When executing tasks, the task manager uses its task slots to isolate and manage the execution of tasks. Each task slot represents a subset of the task manager's resources and can independently execute one or more subtasks. The number of task slots determines the number of tasks that the task manager can execute in parallel.

[0056] This collaboration between the job manager and the task manager ensures that Flink jobs can run efficiently and reliably in a distributed environment while providing good fault tolerance and high availability.

[0057] S308, after the task is executed, the task manager feeds back the execution time and various data generated during the execution process to the job manager;

[0058] S310, collecting task data generated during the execution of the current round of tasks by the task manager, and transmitting the collected first group of task data to the target prediction model;

[0059] Before using the task manager to execute the current round of tasks, the task manager is also used to execute the previous round of tasks, and the task data generated during the execution of the previous round of tasks is input into the target prediction model to obtain the target prediction weight of the predicted execution time and resource allocation of the current round of tasks.

[0060] The target prediction weight is usually used to measure the resource requirements and importance of tasks. The prediction weight can assist the resource scheduler (also known as the resource manager) in deciding how to allocate limited resources to ensure that critical tasks can obtain sufficient resources to meet their needs while also maximizing the overall utilization of resources.

[0061] In the process of generating prediction weights using the target prediction model, the following factors may be considered, but are not limited to:

[0062] 1) Computational complexity of the task: Computationally intensive tasks may require more CPU resources and therefore may be given a higher weight;

[0063] 2) Data throughput requirements: Tasks that need to process large amounts of data may require more memory and storage resources and may therefore be given higher weights;

[0064] 3) Quality of Service (QoS) requirements: Delay-sensitive tasks may need to be allocated resources first to ensure that they can meet the quality of service requirements;

[0065] 4) Task Priority: Some tasks may be more important than others, so they may be given a higher weight to ensure that they receive resources first.

[0066] The resource manager dynamically allocates resources based on the predicted weights and the actual needs of the tasks. This process usually involves complex algorithms and strategies to achieve the optimal allocation of resources. For example, the ResourceManager in Flink is responsible for the provision, recovery, and allocation of resources. It manages task slots, which are the units of resource scheduling in Flink. By adjusting the number of task slots, users can define how subtasks are isolated from each other, thereby affecting the allocation of resources and the execution of tasks.

[0067] In actual distributed systems, resource scheduling is a dynamic and complex process that requires consideration of multiple factors, including task characteristics, resource availability, and system load. Effective resource scheduling can improve the performance and stability of distributed stream processing systems, while also improving resource utilization and cost-effectiveness.

[0068] S312, feeding back the first prediction weight output by the target prediction model to the job manager, and allocating target resources required for the execution process of the next round of tasks based on the first prediction weight.

[0069] The first prediction weight is a prediction weight for resource allocation of the next round of tasks obtained by inputting the first set of task data and the target prediction weight into the target prediction model.

[0070] S314, feeding back the execution result or calculation result of each round of tasks to the Flink program through the job manager.

[0071] By adopting the above method, in the process of executing the current round of tasks, based on the first set of task data generated in the process of executing the current round of tasks and the target prediction weight of the resource configuration output by the target prediction model in the previous round, the target resources that can meet its needs are dynamically allocated in real time, which solves the problem of large errors caused by manual assessment of task volume for resource allocation, ensures the rationality of resource allocation, and achieves the technical effect of improving the accuracy of resource allocation in the process of executing tasks using a distributed stream processing system.

[0072] In an exemplary embodiment, the obtaining of the first set of task data from the distributed stream processing system includes:

[0073] The current round of tasks is transmitted to the job manager through the presentation layer state conversion interface provided by the distributor of the distributed stream processing system;

[0074] Based on the job manager, the job graph of the current round of tasks is decomposed into executable task execution graphs;

[0075] Allocate execution resources for the current round of tasks based on the target resource configuration parameters indicated by the target prediction weights, where the execution resources are the total resources required to execute each execution graph in the task execution graph;

[0076] In the process of executing a current round of tasks based on the execution resources, a first set of task data is obtained.

[0077] like Figure 3 As shown, the client (APP or WEB) submits the job or task to the job manager through the REST interface provided by the dispatcher, where the REST interface is an interface provided by the dispatcher in Flink, which is used to submit jobs and manage running jobs (or tasks).

[0078] The REST interface allows users to interact with the Flink cluster through HTTP requests, including submitting new jobs, canceling jobs, triggering checkpoints, etc. The distributor is an HTTP access point for the cluster and can run across jobs. It starts the job when it receives a job submission request and hands the job to a JobManager for processing.

[0079] The distributor's REST API provides a variety of functions, such as obtaining job configuration, logs, status, and triggering job save points, etc. These APIs make interaction with the Flink cluster more flexible and convenient, and also support interaction with the Flink cluster programmatically.

[0080] For specific REST interface details and available operations, please refer to Flink's official documentation, which contains detailed API descriptions and usage examples. Through these REST interfaces, users can effectively manage and monitor the execution of Flink jobs.

[0081] The job manager is started through the distributor, and the job (including JobGraph) is submitted to the JobMaster. The JobMaster first disassembles the JobGraph (job graph) into an executable Graph (execution graph or task execution graph), and then queries and receives the prediction weight in the output result of the target prediction model. In the case of the query that there is a target prediction weight, the corresponding execution resource is requested from the resource manager according to the target resource configuration parameters indicated by the target prediction weight; if the target prediction weight is not queried, the corresponding default execution resource is requested from the resource manager according to the default resource configuration parameters. The implementation process of requesting execution resources according to the default resource configuration parameters will be described in detail below in conjunction with specific embodiments.

[0082] It should be noted that in the process of executing tasks using the execution resources allocated for the current round of tasks, a monitoring task will be started. Through this monitoring task, some task data (or monitoring indicators) during the task execution process will be collected, such as operator processing speed, task running time, checkpoint health, number of task restarts, JVM and container indicators, etc. The collected task data will be imported into the target prediction model, and the model will be trained to output the prediction indicator and recommended parameters (that is, prediction weight and prediction execution time).

[0083] Through the collaboration mechanism between the job manager and the task manager in the above manner, it is ensured that Flink jobs can run efficiently and reliably in a distributed environment, while providing good fault tolerance and high availability.

[0084] In an exemplary embodiment, the target resource configuration parameters indicated by the target prediction weights are used to allocate execution resources for the current round of tasks, including:

[0085] Based on the target resource configuration parameter, a resource request is sent to a resource manager in the distributed stream processing system, wherein the resource request indicates that the resource quantity indicated by the target resource configuration parameter is expected to be obtained from the resource manager;

[0086] Based on the resource request, a target number of resources is obtained from the resource manager, and the target number of resources is determined as the execution resources for the current round of task allocation.

[0087] Combined with the description in the above embodiment, it can be seen that after the job manager is started through the distributor and the job (including JobGraph) is submitted to the JobMaster, the JobMaster first disassembles the JobGraph (job graph) into an executable Graph (execution graph or task execution graph), and then queries and receives the prediction weight in the output result of the target prediction model. In the case of querying the existence of the target prediction weight, the corresponding execution resource is requested from the resource manager according to the target resource configuration parameters indicated by the target prediction weight.

[0088] Among them, after the task manager is started, it registers its available slots (task slots or task slots) in the resource manager. The resource manager notifies the task manager of the executable resources required to perform the task. The task manager connects to the job manager and provides its own executable resources. The job manager distributes the tasks to be executed to the task manager, and the task manager executes the tasks.

[0089] Obviously, the task manager and the job manager can exchange data, and start a monitoring task for each task manager to collect some task data or monitoring indicators generated during the task execution process, such as operator processing speed, task running time, number of task restarts, etc., and import the data into the target prediction model to train the model.

[0090] It should be noted that in actual application, one task manager may be used to execute one round of tasks, or two or more task managers connected in series may be used to execute the same round of tasks. This embodiment of the present application does not limit this.

[0091] In this embodiment, the target resource configuration parameters indicated by the target prediction weights output by the target prediction model are used to evaluate the resources required to execute the next round of tasks, and a preset number of task slots are automatically configured for them based on the evaluation results, thereby avoiding business anomalies caused by idle resources or task scheduling configuration exceeding the existing resource carrying capacity.

[0092] In addition, compared with the prior art in which the number of task slots and the job scheduling mode are manually configured, in which the resource utilization rate cannot be fully utilized due to the inaccurate estimation of resource consumption caused by manual evaluation of the size of the job, thus leading to the problem of idle resources, the resource allocation method in this embodiment is more reasonable based on the prediction weights dynamically output in real time by the target prediction model, which improves the accuracy of the resource allocation results.

[0093] In an exemplary embodiment, the method of obtaining a first set of task data from a distributed stream processing system further includes:

[0094] The current round of tasks is transmitted to the job manager through the presentation layer state conversion interface provided by the distributor of the distributed stream processing system;

[0095] Based on the job manager, the job graph of the current round of tasks is decomposed into executable task execution graphs;

[0096] Obtaining pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters pre-set during the development of the distributed stream processing system;

[0097] Allocate execution resources for the current round of tasks based on default resource configuration parameters;

[0098] Based on the execution resources, the current round of tasks is executed and the first set of task data is generated.

[0099] Combined with the description in the above embodiment, it can be known that the job manager is started through the distributor, and the job (including JobGraph) is submitted to the JobMaster. The JobMaster first disassembles the JobGraph (job graph) into an executable Graph (execution graph or task execution graph), and then queries and receives the prediction weight in the output result of the target prediction model. If the target prediction weight is not queried, the corresponding default execution resource is requested from the resource manager according to the default resource configuration parameters, and the execution resources are allocated to the current round of tasks according to the default resource configuration parameters, so as to execute the current round of tasks.

[0100] The specific steps for executing Flink tasks using the default resource configuration parameters are as follows:

[0101] S11, the client (APP or WEB) submits the job to the JobManager through the REST interface provided by the dispatcher;

[0102] S12, the distributor starts the JobMaster and submits the job (including the JobGraph) to the JobMaster;

[0103] S13, JobMaster first disassembles JobGraph into executable Graph and requests the corresponding execution resource (slot) from the resource manager;

[0104] S14, determine whether the current resource manager has sufficient resources, if not, start a new TaskManager;

[0105] S15, after TaskManager is started, it registers its available slots with the resource manager;

[0106] S16, the resource manager notifies the TaskManager job to provide executable resources;

[0107] S17, TaskManager connects to JobMaster and provides its own executable resources;

[0108] S18, TaskManager connects to JobMaster and provides its own executable resources;

[0109] S19, TaskManager executes tasks and can exchange data with each other;

[0110] S20, start the monitoring item and provide a UI for indicator viewing.

[0111] Through the pre-configured default resource configuration parameters, it can be ensured that when the monitoring task is not started, execution resources can still be allocated to the current round of tasks according to the default configuration, which ensures the execution process of the task and ensures that the distributed stream processing system can adapt to the changing workload and environmental conditions, thereby improving the stability of the system.

[0112] In an exemplary embodiment, the first predicted execution time and the first predicted weight of the resource allocation for the next round of tasks are obtained by inputting the first set of task data and the target prediction weight into the target prediction model, including:

[0113] Obtaining a second predicted execution time and a second predicted weight output by the target prediction model, wherein the second predicted weight represents the task attributes of a previous round of tasks predicted by the target prediction model and the priority of a resource allocation request when allocating resources required for executing the previous round of tasks, wherein the previous round of tasks is a round of tasks executed before executing the current round of tasks;

[0114] Obtain a second set of task data from the distributed stream processing system, where the second set of task data is used to describe task information of a previous round of tasks;

[0115] By inputting the second set of task data and the second prediction weight into the target prediction model, a target prediction weight and a target prediction execution time are obtained;

[0116] By inputting the first set of task data and the target prediction weight into the target prediction model, a first predicted execution time and a first predicted weight are obtained.

[0117] In the embodiments of the present application, the H20.AI machine learning model platform can be used as the above-mentioned target prediction model but is not limited to it. The model platform provides an interface for integration with Flink. By designing the model and using the gradient boosting learning technology to import the execution time of Flink tasks, the execution time of future tasks can be regressed and predicted. By importing the number of failures and failure nodes of Flink task checkpoints through the decision tree learning technology, the weight of resource allocation for future tasks can be judged and predicted.

[0118] Obviously, the input of the target prediction model includes but is not limited to two parts. The first part is the task data of the current round of tasks being executed, and the second part is the target prediction weight of the current round of tasks output by the target prediction model based on the task data collected during the execution of the previous round of tasks; the output of the target prediction model includes the first predicted execution time and the first predicted weight of the next round of tasks.

[0119] That is to say, the method of obtaining the target prediction weight of the current round of tasks includes: obtaining the task data of the previous round of tasks, and obtaining the second prediction execution time and the second prediction weight output by the target prediction model; by inputting the second set of task data and the second prediction weight into the target prediction model, the target prediction weight and target prediction execution time of the current round of tasks are obtained.

[0120] After obtaining the target prediction weight output by the target prediction model, the first set of task data and the target prediction weight are input into the target prediction model to obtain the first prediction execution time and the first prediction weight output by the model.

[0121] That is to say, by using the prediction weights output by the target prediction model in the previous round of training and the task data of the current round of tasks as the input of the model in the current round of training, the target prediction model is continuously trained and optimized, so that the monitoring data after each task execution is fed back to the target prediction model to further train and optimize the target prediction model, thereby improving the prediction accuracy and stability of the model.

[0122] At the same time, by collecting system monitoring data in real time and using machine learning models for prediction, accurate adjustment of dynamic resource requests is achieved, effectively improving the resource utilization and job processing efficiency of the distributed stream processing system. The actual results show that after adopting the technical solution in the embodiment of the present application, the waste of system resources is significantly reduced, the job processing delay is significantly reduced, and the overall performance is significantly improved. In addition, the continuous training and optimization mechanism of the model ensures that the system can adapt to the ever-changing workload and environmental conditions, improves the stability and flexibility of the system, and provides strong support for large-scale data processing.

[0123] In an exemplary embodiment, after obtaining the first predicted execution time and the first predicted weight by inputting the first set of task data and the target prediction weight into the target prediction model, the method further includes:

[0124] generating a task execution time curve based on the first predicted execution time and the second predicted execution time;

[0125] When the task execution time curve indicates that N consecutive predicted execution times tend to the target equilibrium point, determining that the performance of the target prediction model meets the preset requirements, where N is a positive integer greater than or equal to 1; or

[0126] generating a task execution time curve based on the first predicted execution time and the second predicted execution time;

[0127] Determine a rate of change of the task execution time curve based on the N predicted execution times in the task execution time curve;

[0128] When the rate of change of the task execution time curve is less than or equal to a preset threshold, determining that the performance of the target prediction model meets the preset requirements; or

[0129] generating a task execution time curve based on the first predicted execution time and the second predicted execution time;

[0130] Determine a rate of change of the task execution time curve based on the N predicted execution times in the task execution time curve;

[0131] When the task execution time curve indicates that N consecutive predicted execution times tend to the target equilibrium point and the rate of change is less than or equal to a preset threshold, it is determined that the performance of the target prediction model meets the preset requirements.

[0132] In combination with the description in the above embodiments, it can be seen that after each task is executed, the target prediction model will be trained in real time using the generated task data. So how to evaluate the accuracy of the model prediction and how to ensure that the model can be adaptively adjusted for different business scenarios, the following possible evaluation methods are given in the embodiments of this application.

[0133] For example, after each task is executed, the target prediction model will not only generate the predicted weights for the resource configuration of the next round of tasks, but also generate the predicted execution time for the next round of tasks, and record the predicted execution time output by each training process. Then, based on a set of predicted execution times, it will evaluate whether the training results of the model tend to be more accurate in the direction of the resource configuration results output next time.

[0134] Specifically, when the lengths of N consecutive prediction execution times tend toward the target balance point, it can be determined that the training process of the target prediction model is continuously optimized in the direction of outputting a better resource allocation result next time, and the performance of the target prediction model can meet the preset requirements.

[0135] For another example, by using the task execution time curve generated by the recorded N predicted execution times, the change rate of the task time curve is determined, where the change rate of the time curve usually refers to the volatility or stability of the time series data. If the change rate of the time curve is small, it usually means that the change between data points is not large, or the volatility of the data is small, which can be interpreted as better stability of the time series, or that the time series is more stable.

[0136] However, it should be noted that a small rate of change does not necessarily mean that the time series is completely stable. Therefore, this embodiment also proposes another method for evaluating the model training results, that is, when the N consecutive task execution times represented by the task execution time curve tend to the target balance point and the rate of change of the task execution time curve is less than or equal to the preset threshold, it is determined that the performance of the target prediction model meets the preset requirements.

[0137] Therefore, through the above evaluation method, we ensure that the training process of the target prediction model is developing in the direction of being able to output better resource allocation results next time, and use a more optimized target prediction model to predict the resource requirements of subsequent tasks. Specifically, before task scheduling, the number of slots requested from the resource manager is dynamically adjusted according to the prediction results of the target prediction model, thereby dynamically adjusting the execution resources allocated to the next round of tasks, thereby improving the accuracy of resource allocation.

[0138] In addition, after each task is executed, the prediction ability of the target prediction model can be continuously trained and optimized by feeding the collected monitoring data (task data) back to the target prediction model. This continuous training and optimization mechanism ensures that the distributed stream processing system can adapt to changing workloads and environmental conditions, thereby improving the stability of the system.

[0139] In another optional embodiment, after obtaining the first set of task data from the distributed stream processing system, the method further includes:

[0140] Performing a modification operation on the first set of task data to obtain a modified third set of task data;

[0141] By inputting the third set of task data and the target prediction weight into the target prediction model, the third predicted execution time and the third predicted weight of the next round of tasks are obtained.

[0142] In order to ensure that different training can be performed for different businesses, in the actual application process, the data collected by the current target prediction model can be added or deleted for different businesses, and the input data of the model can be flexibly determined by using the task data after addition or deletion, thereby adjusting the training direction of the model, so that the final prediction result and the prediction weight of the resource configuration include the computing and storage resources used by each node.

[0143] Furthermore, the physical resources of the production environment can be allocated in the most reasonable way, which improves the flexible configuration of the target prediction model and enables the target prediction model to adapt to different business scenarios, ensuring the smooth execution of tasks.

[0144] From the description of the above embodiments, it can be seen that the technical solution of the present application uses the task data generated during the execution of existing tasks and inputs it into the target prediction model, so that the target prediction model gives the recommended scheduling strategy and resource allocation. In other words, by reusing the collected task data, the process can be optimized for the existing data without increasing the resource consumption.

[0145] In addition, based on the task data collected during the task execution, the model obtained in each round of training is more accurate and efficient than the resource allocation given by the task experience, which reduces the labor cost and realizes the automatic task allocation. Furthermore, the performance and accuracy of the target prediction model will be improved synchronously with the increase of the number of task executions. Therefore, the efficiency and accuracy of task scheduling using the improved distributed stream processing system in the embodiment of the present application will also continue to grow positively.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0147] In this embodiment, a resource allocation device is also provided to implement the above-mentioned embodiments and preferred implementation modes, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the modules described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0148] Figure 4 is a structural block diagram of a resource allocation device according to an embodiment of the present application, the device comprising:

[0149] A first acquisition unit 402 is used to acquire a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks;

[0150] A first processing unit 404 is configured to obtain a first predicted execution time and a first predicted weight of resource allocation for a next round of tasks by inputting a first set of task data and a target prediction weight into a target prediction model, wherein the target prediction weight is used to represent the task attributes of a current round of tasks and the priority of resource allocation requests when allocating resources required for executing the current round of tasks;

[0151] The second processing unit 406 is used to allocate target resources required in the execution process for the next round of tasks based on the first predicted execution time and the first predicted weight.

[0152] In an exemplary embodiment, the above-mentioned first acquisition unit 402 includes: a first transmission module, which is used to transmit the current round of tasks to the job manager through the presentation layer state transition interface provided by the distributor of the distributed stream processing system; a first disassembly module, which is used to disassemble the job graph of the current round of tasks into executable task execution graphs based on the job manager; a first allocation module, which is used to allocate execution resources for the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight, wherein the execution resources are the total resources required to execute each execution graph in the task execution graph; a first acquisition module, which is used to obtain a first set of task data in the process of executing the current round of tasks based on the execution resources.

[0153] In an exemplary embodiment, the above-mentioned first allocation module includes: a sending sub-module, used to send a resource request to a resource manager in a distributed stream processing system based on a target resource configuration parameter, wherein the resource request indicates the expected number of resources indicated by the target resource configuration parameter to be obtained from the resource manager; a first acquisition sub-module, used to obtain a target number of resources from the resource manager based on the resource request, and determine the target number of resources as the execution resources for the current round of task allocation.

[0154] In an exemplary embodiment, the above-mentioned first acquisition unit 402 includes: a second transmission module, which is used to transmit the current round of tasks to the job manager through the presentation layer state conversion interface provided by the distributor of the distributed stream processing system; a second disassembly module, which, based on the job manager, disassembles the job graph of the current round of tasks into an executable task execution graph; a second acquisition module, which is used to obtain pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters pre-set during the development of the distributed stream processing system; a second allocation module, which is used to allocate execution resources for the current round of tasks based on the default resource configuration parameters; a first processing module, which is used to execute the current round of tasks based on the execution resources and generate a first set of task data.

[0155] In an exemplary embodiment, the above-mentioned first processing unit 404 includes: a third acquisition module, used to obtain the second predicted execution time and the second predicted weight output by the target prediction model, wherein the second predicted weight represents the task attributes of the previous round of tasks predicted by the target prediction model and the priority of the resource allocation request when ensuring the allocation of resources required for the execution of the previous round of tasks, and the previous round of tasks is a round of tasks executed before the execution of the current round of tasks; a fourth acquisition module, used to obtain a second set of task data from the distributed stream processing system, wherein the second set of task data is used to describe the task information of the previous round of tasks; by inputting the second set of task data and the second predicted weight into the target prediction model, the target prediction weight and the target predicted execution time are obtained; a second processing module, used to obtain the first predicted execution time and the first prediction weight by inputting the first set of task data and the target prediction weight into the target prediction model.

[0156] In an exemplary embodiment, the above-mentioned device also includes: a third processing unit, which is used to generate a task execution time curve based on the first predicted execution time and the second predicted execution time after obtaining the first predicted execution time and the first predicted weight by inputting the first group of task data and the target prediction weight into the target prediction model; when the task execution time curve indicates that N consecutive predicted execution times tend to the target balance point, determine that the performance of the target prediction model meets the preset requirements, wherein N is a positive integer greater than or equal to 1; or generate a task execution time curve based on the first predicted execution time and the second predicted execution time; determine the change rate of the task execution time curve based on the N predicted execution times in the task execution time curve; when the change rate of the task execution time curve is less than or equal to a preset threshold, determine that the performance of the target prediction model meets the preset requirements; or generate a task execution time curve based on the first predicted execution time and the second predicted execution time; determine the change rate of the task execution time curve based on the N predicted execution times in the task execution time curve; when the task execution time curve indicates that N consecutive task execution times tend to the target balance point and the change rate is less than or equal to the preset threshold, determine that the performance of the target prediction model meets the preset requirements.

[0157] In an exemplary embodiment, the above-mentioned device also includes: a fourth processing unit, which is used to modify the first group of task data after obtaining the first group of task data from the distributed stream processing system to obtain a modified third group of task data; a fifth processing unit, which is used to obtain a third predicted execution time and a third predicted weight for the next round of tasks by inputting the third group of task data and the target prediction weight into a target prediction model.

[0158] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0159] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0160] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:

[0161] S1, obtaining a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks;

[0162] S2, obtaining a first predicted execution time and a first predicted weight of resource allocation for a next round of tasks by inputting the first set of task data and the target prediction weight into a target prediction model, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when ensuring the allocation of resources required for the execution of the current round of tasks;

[0163] S3: Allocate target resources required in the execution process of the next round of tasks based on the first predicted execution time and the first predicted weight.

[0164] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0165] The embodiment of the present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0166] Optionally, in this embodiment, the processor 504 may be configured to perform the following steps through a computer program:

[0167] S1, obtaining a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks;

[0168] S2, obtaining a first predicted execution time and a first predicted weight of resource allocation for a next round of tasks by inputting the first set of task data and the target prediction weight into a target prediction model, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when ensuring the allocation of resources required for the execution of the current round of tasks;

[0169] S3: Allocate target resources required in the execution process of the next round of tasks based on the first predicted execution time and the first predicted weight.

[0170] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0171] Alternatively, a person skilled in the art may understand that: Figure 5 The structure shown is for illustration only. Figure 5 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 5 Different configurations are shown.

[0172] Among them, the memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the resource allocation method and resource allocation device in the embodiments of the present application. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, that is, realizing the above-mentioned resource allocation method. The memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include a memory remotely arranged relative to the processor 504, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 502 can be specifically, but not limited to, used to store a first set of task data, target prediction weights, and a first prediction execution time, etc. As an example, such as Figure 5 As shown, the memory 502 may include but is not limited to the first acquisition unit 402, the first processing unit 404 and the second processing unit 406 in the resource allocation device. In addition, other module units in the resource allocation device may also be included but are not limited to, which will not be repeated in this example.

[0173] Optionally, the transmission device 506 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0174] In addition, the electronic device mentioned above further includes: a display 508; and a connection bus 510, which is used to connect various module components in the electronic device mentioned above.

[0175] In other embodiments, the electronic device may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0176] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0177] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0178] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0179] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0180] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A resource allocation method, characterized in that: include: Acquire a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks; By inputting the first group of task data and the target prediction weight into the target prediction model, a first predicted execution time and a first predicted weight of resource allocation for the next round of tasks are obtained, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating the resources required for executing the current round of tasks; Based on the first predicted execution time and the first predicted weight, target resources required in the execution process are allocated to the next round of tasks.

2. The method according to claim 1, characterized in that The obtaining of a first set of task data from the distributed stream processing system includes: Transmitting the current round of tasks to a job manager through a presentation layer state conversion interface provided by a distributor of the distributed stream processing system; Based on the job manager, the job graph of the current round of tasks is decomposed into executable task execution graphs; Allocate execution resources for the current round of tasks based on the target resource configuration parameters indicated by the target prediction weight, wherein the execution resources are the total resources required to execute each execution graph in the task execution graph; In the process of executing the current round of tasks based on the execution resources, the first set of task data is obtained.

3. The method according to claim 2, characterized in that The allocating execution resources for the current round of tasks based on the target resource configuration parameters indicated by the target prediction weights includes: Based on the target resource configuration parameter, sending a resource request to a resource manager in the distributed stream processing system, wherein the resource request indicates that the resource quantity indicated by the target resource configuration parameter is expected to be obtained from the resource manager; Based on the resource request, a target amount of resources is obtained from the resource manager, and the target amount of resources is determined as the execution resources for the current round of task allocation.

4. The method according to claim 1, characterized in that: The step of obtaining a first set of task data from the distributed stream processing system further includes: Transmitting the current round of tasks to a job manager through a presentation layer state conversion interface provided by a distributor of the distributed stream processing system; Based on the job manager, the job graph of the current round of tasks is decomposed into executable task execution graphs; Acquire pre-configured default resource configuration parameters, wherein the default resource configuration parameters are parameters pre-set during the development of the distributed stream processing system; Allocating execution resources for the current round of tasks based on the default resource configuration parameters; Based on the execution resources, the current round of tasks is executed, and the first set of task data is generated.

5. The method according to claim 1, characterized in that The step of obtaining the first predicted execution time and the first predicted weight of the resource allocation for the next round of tasks by inputting the first group of task data and the target prediction weight into the target prediction model comprises: Obtaining a second predicted execution time and a second predicted weight output by the target prediction model, wherein the second predicted weight represents the task attributes of a previous round of tasks predicted by the target prediction model and the priority of a resource allocation request when allocating resources required for executing the previous round of tasks, wherein the previous round of tasks is a round of tasks executed before executing the current round of tasks; Acquire a second set of task data from the distributed stream processing system, wherein the second set of task data is used to describe task information of the previous round of tasks; By inputting the second set of task data and the second prediction weight into the target prediction model, the target prediction weight and the target prediction execution time are obtained; The first predicted execution time and the first predicted weight are obtained by inputting the first set of task data and the target prediction weight into a target prediction model.

6. The method according to claim 5, characterized in that After obtaining the first predicted execution time and the first predicted weight by inputting the first set of task data and the target prediction weight into a target prediction model, the method further includes: generating a task execution time curve based on the first predicted execution time and the second predicted execution time; When the task execution time curve indicates that N consecutive predicted execution times all tend toward the target equilibrium point, determining that the performance of the target prediction model meets the preset requirements, wherein N is a positive integer greater than or equal to 1; or generating the task execution time curve based on the first predicted execution time and the second predicted execution time; Determining a rate of change of the task execution time curve based on the N predicted execution times in the task execution time curve; When the rate of change of the task execution time curve is less than or equal to a preset threshold, determining that the performance of the target prediction model meets the preset requirement; or generating the task execution time curve based on the first predicted execution time and the second predicted execution time; Determining a rate of change of the task execution time curve based on the N predicted execution times in the task execution time curve; When the task execution time curve indicates that N consecutive task execution times all tend toward the target balance point and the rate of change is less than or equal to the preset threshold, it is determined that the performance of the target prediction model meets the preset requirements.

7. The method according to any one of claims 1 to 6, characterized in that After obtaining the first set of task data from the distributed stream processing system, the method further includes: Performing a modification operation on the first set of task data to obtain a modified third set of task data; By inputting the third group of task data and the target prediction weight into the target prediction model, a third predicted execution time and a third predicted weight of the next round of tasks are obtained.

8. A resource allocation device, characterized in that: include: A first acquisition unit, configured to acquire a first set of task data from the distributed stream processing system, wherein the first set of task data is used to describe task information of a current round of tasks; A first processing unit is configured to obtain a first predicted execution time and a first predicted weight of resource allocation for the next round of tasks by inputting the first group of task data and the target prediction weight into a target prediction model, wherein the target prediction weight is used to represent the task attributes of the current round of tasks and the priority of resource allocation requests when allocating the resources required for executing the current round of tasks; The second processing unit is used to allocate target resources required in the execution process of the next round of tasks based on the first predicted execution time and the first predicted weight.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images