Resource requirement determination method, resource allocation method, and electronic device

Abstract

Embodiments of the present disclosure provide a resource requirement determination method, a resource allocation method, and an electronic device. The resource requirement determination method comprises: determining one or more sub-tasks included in an offline workflow; for any target sub-task among the one or more sub-tasks, on the basis of description information of the target sub-task, determining a preset number of historical sub-tasks similar to the target sub-task, wherein the description information is used to characterize attributes of the target sub-task and a dependency relationship between the target sub-task and other sub-tasks; and on the basis of actual resource consumption values of the preset number of historical sub-tasks, determining a resource requirement of the target sub-task, and, on the basis of the resource requirement of the target sub-task, generating and running the target sub-task. The method avoids excessively long offline workflow runtime, offline workflow failures, or resource waste caused by improper resource requirement settings.

Description

[0001] Resource Demand Determination Method, Resource Allocation Method, and Electronic Device. This disclosure claims priority to Chinese patent application number 202410384555.4, filed with the China Patent Office on March 29, 2024, and entitled "Resource Demand Determination Method, Resource Allocation Method, and Electronic Device," the entire contents of which are incorporated herein by reference. Technical Field: This disclosure relates to the field of elastic computing, and more specifically, to a resource demand determination method, a resource allocation method, and an electronic device. Background: Application systems often require the execution of a large number of offline workflows. The development and maintenance of offline workflows involves both offline workflow developers and platform operators. Developers are responsible for developing the business code for offline workflows, while platform operators are responsible for maintaining the offline workflows running in container clusters to ensure their normal operation. When running an offline workflow, it's necessary to set its resource requirements, including CPU and memory resource requirements. However, neither developers nor platform operators can guarantee accurate setting of these requirements, which can easily lead to excessive offline workflow runtimes, failures, or resource waste. SUMMARY OF THE INVENTION The present disclosure provides a resource requirement determination method, a resource allocation method, and an electronic device to prevent excessive offline workflow runtimes, failures, or resource waste caused by improper resource requirement settings. In a first aspect, the present disclosure provides a method for determining resource requirements for an offline workflow, comprising: determining one or more subtasks included in the offline workflow; for any target subtask among the one or more subtasks, determining a preset number of historical subtasks similar to the target subtask based on description information of the target subtask, wherein the description information characterizes the attributes of the target subtask and its dependencies with other subtasks; determining the resource requirements of the target subtask based on the actual resource consumption values ​​of the preset number of historical subtasks, and generating and running the target subtask based on the resource requirements of the target subtask.In a second aspect, the present disclosure provides a resource allocation method, comprising: obtaining a resource requirement of a target subtask of an offline workflow, wherein the offline workflow includes one or more subtasks, the target subtask being any subtask of the offline workflow, the resource requirement of the target subtask being determined based on actual resource consumption values ​​of a preset number of historical subtasks similar to the target subtask, the preset number of historical subtasks being determined based on description information of the target subtask, the description information being used to characterize attributes of the target subtask and dependencies between the target subtask and other subtasks; and allocating resources to the target subtask based on the resource requirement of the target subtask, so as to generate and run the target subtask. In a third aspect, the present disclosure provides a resource requirement determination device, comprising: a splitting module for determining one or more subtasks included in an offline workflow; a matching module for determining, for any target subtask among the one or more subtasks, a preset number of historical subtasks similar to the target subtask based on the target subtask's description information, wherein the description information characterizes the target subtask's attributes and its dependencies with other subtasks; and a determination module for determining the target subtask's resource requirements based on the actual resource consumption values ​​of the preset number of historical subtasks, and generating and executing the target subtask based on the target subtask's resource requirements. In a fourth aspect, the present disclosure provides an electronic device, comprising: a memory and a processor; the memory for storing a computer program; the processor for executing the computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the method according to the first or second aspect. In a fifth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer program is stored in the computer medium, and when executed by the processor, causes the processor to perform the method according to the first or second aspect. In a sixth aspect, the present disclosure provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect or the second aspect is implemented.The resource requirement determination method, resource allocation method, and electronic device provided herein determine the resource requirements of one or more subtasks contained in an offline workflow. For each target subtask, a preset number of historical subtasks similar to the target subtask are determined from historical subtasks. The actual resource consumption values ​​of these preset number of historical subtasks are used as a reference to determine the resource requirements of the target subtask. This ensures that the resource requirements of the target subtask meet the actual resource consumption requirements of the target subtask, thus avoiding excessive offline workflow runtimes, offline workflow failures, or resource waste caused by resource requirements not matching the actual resource consumption of the target subtask. BRIEF DESCRIPTION OF THE DRAWINGS To more clearly illustrate the embodiments of the present disclosure or technical solutions in the prior art, the following briefly introduces the drawings required in the embodiments or descriptions of the prior art. Obviously, the drawings described below represent some embodiments of the present disclosure. Persons skilled in the art can derive other drawings based on these drawings without inventive effort. Figure 1 is a schematic diagram of an offline workflow provided by an embodiment of the present disclosure; Figure 2 is a schematic diagram of a workflow scheduling and execution system provided by an embodiment of the present disclosure; Figure 3 is a flowchart of a resource requirement determination method provided by an embodiment of the present disclosure; Figure 4 is a flowchart of a resource allocation method provided by an embodiment of the present disclosure; Figure 5 is a schematic diagram of the structure of a resource requirement generation device provided by an embodiment of the present disclosure; and Figure 6 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS To further clarify the objectives, technical solutions, and advantages of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a portion of the embodiments of the present disclosure, but not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without inventive effort are within the scope of protection of the present disclosure. The following describes the terms used in the embodiments of the present disclosure. Offline workflow: An independently executed computing task, typically used to process large amounts of data, perform automated operations, run computing tasks, etc. A workflow can be divided into one or more interdependent subtasks, and typically runs for several minutes to several hours.

[0002] Kubernetes cluster: Container orchestration and operation system. Resource requirements: The hardware resource requirements of computing tasks, including CPU and memory. When running containers in a Kubernetes cluster, resource requirements must be specified. Workflow controller: Parses the definition of offline workflows and generates subtasks to run in the Kubernetes cluster. Workflow resource requirement recommendation system: Includes a subtask resource recommender and a subtask resource collector. The subtask resource collector is used to collect historical data on workflow execution, and the subtask resource recommender is used to construct resource requirements for subtasks. Figure 1 is a schematic diagram of an offline workflow provided by an embodiment of the present disclosure. As shown in Figure 1, the offline workflow includes subtask 1, subtask 2, subtask 3, subtask 4, and subtask 5. When actually running subtasks, each subtask can run in parallel with multiple subtasks. For example, 10 subtasks 2 run in parallel and 20 subtasks 3 run in parallel. In offline workflows, dependencies exist between subtasks. The output of the previous subtask serves as the input for the next subtask. A subtask's resource requirements are related to both the subtask itself and the dependencies between these inputs and outputs. A subtask consists of an image, a script, and input and output parameters. A subtask's image refers to the container image it runs, and a subtask's script represents the computational task within the subtask. A subtask's resource requirements are determined by the workflow in which it resides, the dependencies between subtasks, the image, and the script. Improper resource requirements, for example, setting the CPU resource requirement lower than actual consumption, can cause the offline workflow to be constantly constrained, resulting in long runtimes and low efficiency. Setting the memory resource requirement lower than actual consumption can cause an out-of-memory error during the offline workflow, leading to failure. Furthermore, setting CPU and memory resource requirements far in excess of actual consumption can waste Kubernetes cluster resources and increase costs. To accurately set resource requirements and ensure stable and efficient offline workflow operation, the present disclosure proposes a solution for generating resource requirements for offline workflows. As shown in Figure 2, the workflow scheduling and execution system includes a workflow controller, a subtask resource collector, a subtask resource recommender, and a vector database. The workflow controller parses the offline workflow definition, specifically the subtasks it contains, and controls the generation and execution of a series of subtasks in the Kubernetes cluster.The subtask resource collector collects the actual resource consumption values ​​of each subtask running in the Kubernetes cluster, including the actual CPU and memory resource consumption values, and stores this data as historical data in a vector database. The vector database stores the actual resource consumption values ​​and descriptions of historical subtasks running in the Kubernetes cluster. This description describes the workflows in which these subtasks reside, inter-subtask dependencies, images, scripts, and other information. The subtask resource recommender generates resource requirements for currently running subtasks based on the historical subtask information stored in the vector database and returns these requirements to the workflow controller. This is explained below with reference to an embodiment. Figure 3 is a flow chart of a resource requirement determination method provided in an embodiment of the present disclosure. As shown in Figure 3, the method includes:

[0003] S301. Determine one or more subtasks included in the offline workflow. The offline workflow submitted by the user defines one or more subtasks and the dependencies between the subtasks. For example, if the user submits an offline workflow with the dependency relationship shown in Figure 1, by parsing the offline workflow definition, it can be determined that the offline workflow includes subtasks 1, 2, 3, 4, and 5. It can also be determined that subtasks 2 and 3 are both dependent on subtask 1, subtask 4 is dependent on subtask 2 and 3, and subtask 5 is dependent on subtask 4. After determining the one or more subtasks included in the offline workflow, the order in which the subtasks are generated and executed can be determined. For example, after generating and executing subtask 1, subtasks 2 and 3 are generated and executed. After executing subtasks 2 and 3, subtask 4 is generated and executed. After executing subtask 4, subtask 5 is generated and executed. Before generating each subtask, the resource requirements of the subtask are determined according to the subsequent steps of the embodiment of the present disclosure, so as to generate and run the subtask based on the resource requirements.

[0004] S302. For any target subtask among one or more subtasks, based on the target subtask's description information, determine a preset number of historical subtasks similar to the target subtask. The description information is used to characterize the attributes of the target subtask and the dependencies between the target subtask and other subtasks. In the disclosed embodiment, the description information of each subtask describes the attributes of the subtask itself and the dependencies between the subtask and other subtasks. The subtask's attribute information includes information such as the offline workflow in which the subtask resides, the image and script of the subtask, and other information. The target subtask is any subtask in the offline workflow. The description information of the target subtask describes the offline workflow in which the target subtask resides, the image and script of the target subtask, and the dependencies between the target subtask and other subtasks in the offline workflow in which it resides. Historical subtasks are subtasks that have already run to completion in the Kubernetes cluster. It is understood that historical subtasks also have corresponding description information. The description information of the historical subtask describes the offline workflow in which the historical subtask resides, the image and script of the historical subtask, and the dependencies between the historical subtask and other subtasks in the offline workflow in which it resides. Based on the similarity between the target subtask's description information and the historical subtask's description information, the similarity between the target subtask and the historical subtasks can be determined, and then a preset number of historical subtasks similar to the target subtask can be determined from the historical subtasks. For example, based on the similarity between the target subtask and the historical subtasks, the three historical subtasks with the highest similarity to the target subtask can be determined from the historical subtasks. The similarity between the target subtask's description information and the historical subtask's description information can be determined by performing text matching on the target subtask's description information and the historical subtask's description information, and the similarity between the target subtask and the historical subtask can be determined based on the text matching. Alternatively, the description information can be converted into vector form, and the similarity between the target subtask and the historical subtask can be determined based on the similarity calculation between the vector corresponding to the target subtask's description information and the vector corresponding to the historical subtask's description information.

[0005] S303: Determine the resource requirements of the target subtask based on the real resource consumption values ​​of a preset number of historical subtasks, and generate and execute the target subtask based on the resource requirements of the target subtask. The real resource consumption values ​​of the historical subtasks are the actual CPU resource consumption and memory resource consumption values ​​collected during the execution of the historical subtasks. The real resource consumption values ​​reflect the actual resource usage of the historical subtasks. Because the preset number of historical subtasks are similar to the target subtask, the real resource consumption values ​​of the preset number of historical subtasks can be used as a reference to determine the resource requirements of the target subtask. This ensures that the resource requirements of the target subtask are neither too large nor too small, and that the target subtask generated based on the resource requirements of the target subtask can execute stably and efficiently. In the method for determining resource requirements for an offline workflow provided in the embodiments of the present disclosure, resource requirements are determined based on one or more subtasks contained in the offline workflow. For each target subtask, a preset number of historical subtasks similar to the target subtask are determined from historical subtasks. The actual resource consumption values ​​of these preset number of historical subtasks are used as a reference to determine the resource requirements of the target subtask. This ensures that the resource requirements of the target subtask are neither too large nor too small. This avoids excessive offline workflow runtimes and failures due to underestimating resource requirements, as well as resource waste due to overestimating resource requirements. The method of the embodiments of the present disclosure is further described based on the above embodiments. Optionally, in the embodiments of the present disclosure, the description information of a subtask may include multiple items of the following information: the name of the offline workflow in which the subtask resides, the name of the subtask, the names of other subtasks on which the subtask depends, the name of the subtask's image file, and the subtask's script. These items and the script file are used to describe the offline workflow in which the subtask resides, the dependencies between the subtask and other subtasks in the offline workflow in which it resides, and the subtask's image and script. Accordingly, the description information of the target subtask may include multiple items of the following information: the name of the offline workflow in which the target subtask resides, the name of the target subtask, the names of the subtasks upon which the target subtask depends, the name of the target subtask's image file, and the script of the target subtask. The description information of the historical subtask may include multiple items of the following information: the name of the offline workflow in which the historical subtask resides, the name of the historical subtask, the names of the subtasks upon which the historical subtask depends, the name of the image file of the historical subtask, and the script of the historical subtask. In one implementation, the description information of the target subtask and the description information of the historical subtask are converted into vectors, and the similarity between the target subtask and the historical subtask is determined based on a similarity calculation between the vectors.Specifically, the description information of the target subtask is vectorized to obtain a description vector for the target subtask; the similarity between the description vector of the target subtask and the description vectors of historical subtasks stored in a vector database is determined, and a preset number of historical subtasks are determined based on the similarity. The description information can be vectorized using any method known in the relevant art, which is not limited in the present disclosure. For example, one-hot model (One Hot Model 1), bag of words model (Bag of Words Model 1), term frequency-inverse document frequency (TF-IDF), N-gram model (N-Gram), word-to-vector model (Word2vec), document-to-vector model (Doc2vec), etc. can be used for vectorization. After determining the similarity between the target subtask's description vector and the description vectors of each historical subtask, a preset number of historical subtasks with the highest similarity can be determined. For example, the first preset number of historical subtasks can be determined by sorting them from high to low similarity. Alternatively, the last preset number of historical subtasks can be determined by sorting them from low to high similarity. In the disclosed embodiment, a database storing data in key-value pairs can be pre-established as a vector database. Each piece of data in the vector database includes a key and a corresponding value. The vector database stores the description vectors of each historical subtask. Furthermore, the vector database can also store the actual resource consumption value of each historical subtask. In the vector database, the description vector of each historical subtask can be used as the key, and the actual resource consumption value of the historical subtask can be used as the value. Information about each historical subtask is stored in a key-value format. Among them, the real resource consumption value of the historical subtask can be obtained in the following way: during the execution of the historical subtask, its real resource consumption value is collected at preset time intervals, and after the historical subtask is completed, its description vector and the maximum value of the multiple collected real resource consumption values ​​are stored in the vector database, with the description vector of the historical subtask as the key and the maximum value of the real resource consumption values ​​of the historical subtask as the value, thereby constituting historical data saved in the form of key-value.For example, during the execution of the historical subtask, the real CPU resource consumption value and memory resource consumption value of the historical subtask are collected at a time interval of once per minute. Assuming that the collected real CPU resource consumption values ​​are 50%, 70%, 80%, 70%, and 82%, respectively, and the real memory resource consumption values ​​are 2.3G, 3G, 3.5G, 3.9G, and 4.5G, respectively, 82% and 4.5G are stored in the vector database as the real CPU resource consumption value and real memory consumption value of the historical subtask, respectively. It can be understood that, for a target subtask, after its resource requirements are determined and the target subtask is created using the method of an embodiment of the present disclosure, during the operation of the target subtask, similar to the above, the actual resource consumption value of the target subtask is collected at preset time intervals, and after the target subtask is completed, the description vector of the target subtask and the maximum value of the collected actual resource consumption values ​​of the target subtask are stored in a vector database, wherein the description vector of the target subtask is used as the key and the maximum value of the actual resource consumption values ​​of the target subtask is used as the value. oIn this way, the target subtask's information can serve as historical data, providing a reference for resource requirements of subsequent subtasks. Based on any of the above embodiments, an example is provided to illustrate how to determine the resource requirements of a target subtask based on the actual resource consumption values ​​of a preset number of historical subtasks. Optionally, a weighted summation is performed on the actual resource consumption values ​​of the preset number of historical subtasks, and the resource requirements of the target subtask are determined based on the result of this weighted summation. The actual resource consumption values ​​of these preset number of historical subtasks, which are similar to the target subtask, can all serve as references for the target subtask's resource requirements. The actual resource consumption values ​​of these preset number of historical subtasks are processed according to preset rules, and the resource requirements of the target subtask are determined based on the processing results. The preset rules can be customized. For example, the preset rules can include weighting the actual resource consumption values ​​of a preset number of historical subtasks to improve accuracy. The weighted summation can be performed by assigning different weights to different historical subtasks, for example, based on similarity, with higher weights assigned to higher similarities. Alternatively, the weighted summation can be performed by averaging, where all historical subtasks are given the same weight. For another example, the preset rule may be to select the maximum value from the actual resource consumption values ​​of a preset number of historical subtasks, and determine the resource requirement of the target subtask based on the maximum value among the actual resource consumption values ​​to avoid resource shortages. Optionally, when determining the resource requirement of the target subtask based on the processing results, to prevent the actual resource consumption of the target subtask from exceeding the resource requirement, an appropriate margin may be added based on the processing results as the resource requirement of the target subtask. This margin may be a preset margin value or margin ratio. For example, if the preset number of historical subtasks is three, and the actual memory consumption values ​​of the three historical subtasks are 2.5 GB, 3 GB, and 3.5 GB, respectively, and the average result is 3 GB, then a margin of 1 GB is added, and 4 GB of memory is set as the memory resource requirement of the target subtask. Alternatively, the resource requirement level can be set based on the processing results according to the optional resource requirement levels provided by the system. For example, the optional resource requirement levels provided by the system include 2GB, 4GB, and 8GB of memory, and 1-core, 2-core, and 4-core CPUs. Based on the processing results, a level greater than the processing result is selected as the resource requirement of the target subtask. For example, assuming the preset number of historical subtasks is 3, and the memory resource requirements of the three historical subtasks are all 8GB, but their actual memory consumption values ​​are 2.5GB, 3GB, and 3.5GB, respectively, and the average result is 3GB, then 4GB of memory is selected as the memory resource requirement of the target subtask.In some scenarios, the actual resource consumption of historical subtasks is high. For example, the actual resource consumption of a historical subtask is close to or equal to the resource requirement of the historical subtask, resulting in excessively long runtimes. Therefore, when setting resource requirements for the target subtask, to prevent similar situations from recurring and ensure stable operation of the target subtask, optionally, if the actual resource consumption of a first historical subtask among a preset number of historical subtasks exceeds a first threshold, the resource requirement of the target subtask is determined to be greater than the resource requirement of the first historical subtask. For example, a certain margin value or margin ratio can be added to the resource requirement of the first historical subtask. Alternatively, based on the optional resource requirement levels provided by the system, a resource requirement level greater than the resource requirement of the first historical subtask can be selected as the resource requirement of the target subtask. It should be noted that the first threshold can be set separately for CPU resources and memory resources. The first threshold can be customized based on the actual application scenario and can be a ratio of resource consumption or a specific numerical value. Taking CPU resources as an example, assuming the first threshold for CPU resources is 85%, the CPU resource requirements of three historical subtasks are all 2 cores, and the actual CPU resource consumption values ​​are 70%, 78%, and 100%, respectively. Since there is a first historical subtask whose actual CPU resource consumption value exceeds 85% of the first threshold, 2 cores are added to the CPU resource requirement of 2 cores for the first historical subtask, making 4 cores the CPU resource requirement of the target subtask. Taking memory resources as an example, assuming the first threshold for memory resources is 80%, the memory resource requirements of three historical subtasks are all 4GB, and the actual memory resource consumption values ​​are 2.5GB, 3GB, and 3.8GB, respectively. Since there is a first historical subtask whose actual memory resource consumption value exceeds 80%, the memory resource requirement of the first historical subtask is increased by a first proportion, for example, to 2GB, making 6GB the memory resource requirement of the target subtask. If an out-of-memory error occurs during subtask execution, causing the subtask to fail, the workflow controller will record the failure caused by this error and recreate the subtask. To ensure that the rebuilt subtask can run normally, in the embodiment of the present disclosure, for a target subtask, if the target subtask fails to execute and the reason for the failure is insufficient memory, the memory resource requirement in the resource requirement of the target subtask is increased, and the target subtask is rebuilt and run again.If the target subtask encounters an out-of-memory error, it means the target subtask's memory resource requirement is set too low. Therefore, the memory resource requirement needs to be increased. This can be done by adding a certain margin value or margin ratio to the target subtask's memory resource. Alternatively, based on the optional resource requirement levels provided by the system, a memory resource level greater than the target subtask's can be selected as the target subtask's new resource requirement. For example, if the target subtask's resource requirements include a 2-core CPU resource requirement and a 4GB memory resource requirement, and the target subtask fails due to insufficient memory, the target subtask's resource requirements can be increased to 2-core CPU resource requirements and 6GB memory resource requirements, and the target subtask can be rebuilt and run based on the increased resource requirements. This correction mechanism ensures the target subtask's successful completion. Figure 4 is a schematic flow chart of a resource allocation method provided in an embodiment of the present disclosure. As shown in Figure 4, the method includes:

[0006] S401. Obtain resource requirements for a target subtask of an offline workflow. The offline workflow includes one or more subtasks, and the target subtask is any subtask of the offline workflow. The resource requirements for the target subtask are determined based on actual resource consumption values ​​of a preset number of historical subtasks similar to the target subtask. The preset number of historical subtasks is determined based on description information of the target subtask. The description information is used to characterize attributes of the target subtask and dependencies between the target subtask and other subtasks.

[0007] S402. Allocate resources to the target subtask based on the target subtask's resource requirements to generate and run the target subtask. This method can be performed by a cloud server, for example, a cloud server in a Kubernetes cluster. As shown in FIG2 , the workflow scheduling and execution system determines the target subtask's resource requirements based on the method of the aforementioned embodiment. The cloud server in the Kubernetes cluster obtains the target subtask's resource requirements and, based on the resource requirements, allocates or creates a container for the target subtask to generate and run the target subtask. Because the target subtask's resource requirements determined using the method of the aforementioned embodiment better meet the target subtask's actual consumption, the target subtask is ensured to run stably and efficiently, avoiding excessive offline workflow runtimes, offline workflow failures, and resource waste. FIG5 is a schematic diagram of a resource requirement determination apparatus provided in an embodiment of the present disclosure. As shown in Figure 5, a resource requirement determination device 500 includes: a splitting module 501 for determining one or more subtasks included in the offline workflow; a matching module 502 for determining, for any target subtask among the one or more subtasks, a preset number of historical subtasks similar to the target subtask based on the target subtask's description information; and a determination module 503 for determining the target subtask's resource requirements based on the actual resource consumption values ​​of the preset number of historical subtasks, and generating and running the target subtask based on the target subtask's resource requirements. In one implementation, the description information includes multiple items of the following information: the name of the offline workflow, the name of the target subtask, the names of the subtasks on which the target subtask depends, the name of the target subtask's image file, and the script of the target subtask. In one implementation, the matching module is configured to: vectorize the description information of the target subtask to obtain a description vector for the target subtask; determine the similarity between the description vector of the target subtask and the description vectors of historical subtasks stored in a vector database, and determine a preset number of historical subtasks based on the similarity. In another implementation, the apparatus further includes: a collection module configured to collect the target subtask's actual resource consumption value at preset time intervals during the execution of the target subtask; and after the target subtask completes execution, store the maximum value of the target subtask's description vector and the collected actual resource consumption value of the target subtask in the vector database.In one implementation, the determination module is configured to process the actual resource consumption values ​​of the preset number of historical subtasks according to preset rules and determine the resource requirement of the target subtask based on the processing results. In one implementation, the determination module is configured to determine the resource requirement of the target subtask to be greater than the resource requirement of the first historical subtask if the actual resource consumption value of the first historical subtask among the preset number of historical subtasks exceeds a first threshold. In one implementation, the system further includes a reconstruction module configured to, if the target subtask fails due to insufficient memory, increase the memory resource requirement in the target subtask's resource requirements, rebuild the target subtask, and run it again. In one implementation, the system further includes an interaction module configured to output the resource requirement of the target subtask to a user interface and receive a resource requirement feedback instruction input by a user; and determine or adjust the resource requirement of the target subtask based on the resource requirement feedback instruction. The apparatus of the embodiments of the present disclosure can be used to implement the offline workflow resource requirement determination method described in the aforementioned embodiments. The implementation principles and technical effects are similar and will not be further described here. Figure 6 is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. As shown in Figure 6, the electronic device 600 may include at least one processor 601 for implementing the offline workflow resource requirement determination method provided in an embodiment of the present disclosure. Optionally, the electronic device 600 also includes at least one memory 602 for storing program instructions and / or data. The memory 602 is coupled to the processor 601. Coupling in the embodiments of the present disclosure refers to an indirect coupling or communication connection between devices, units, or modules, which may be electrical, mechanical, or other forms, and is used for information exchange between the devices, units, or modules. The processor 601 may operate in conjunction with the memory 602. The processor 601 may execute program instructions stored in the memory 602. At least one of the at least one memory may be included in the processor. Optionally, the electronic device 600 also includes a communication interface 603 for communicating with other devices via a transmission medium, thereby enabling the electronic device 600 to communicate with other devices. The communication interface 603 may be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of performing transceiver functions. The processor 601 may utilize the communication interface 603 to transmit and receive data and / or information, and to implement the methods provided in the embodiments of the present disclosure. For details, please refer to the detailed description in the previous embodiments and will not be repeated here. The specific connection medium between the processor 601, memory 602, and communication interface 603 is not limited in the embodiments of the present disclosure.In FIG6 , the embodiment of the present disclosure illustrates a connection between a processor 601, a memory 602, and a communication interface 603 via a bus 604. Bus 604 is represented by a bold line in FIG6 , and the connection between other components is for illustrative purposes only and is not intended to be limiting. Such buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG6 illustrates only one bold line, but this does not imply that there is only one bus or only one type of bus. It should be understood that the processor in the embodiment of the present disclosure can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic diagrams disclosed in the embodiments of this disclosure can be implemented or executed. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of this disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules can be located in a storage medium known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in a memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the aforementioned methods. It should also be understood that the memory in the embodiments of this disclosure can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. The present disclosure also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When executed by a processor, the computer executes the method described in any of the aforementioned embodiments. The present disclosure also provides a computer program product comprising the computer program, which, when executed by a processor, implements the method described in any of the aforementioned embodiments. As used in this disclosure, the terms "unit," "module," and the like may be used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented using hardware or software depends on the specific application and design constraints of the technical solution. Professionals may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this disclosure. In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatuses, devices, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of the units is merely one logical functional division. In actual implementation, other divisions may be used, such as combining or integrating multiple units or components into another system, or omitting or not implementing certain features.Furthermore, the coupling, direct coupling, or communication connection shown or discussed may be an indirect coupling or communication connection through interfaces, devices, or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of the present embodiment according to actual needs. Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. In the above embodiments, the functions of each functional unit may be implemented in whole or in part through software, hardware, firmware, or any combination thereof. When implemented using software, they may be implemented in whole or in part in the form of a computer program product. This computer program product includes one or more computer instructions (programs). When these computer program instructions (programs) are loaded and executed on a computer, they fully or partially generate the processes or functions described in the embodiments of the present disclosure. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)). If the function is implemented as a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.Based on this understanding, the technical solution of this disclosure, or the portion that contributes to the prior art, or the portion of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks. The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject. The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope of this disclosure should be covered by the scope of protection of this disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

Claims 1. A method for determining resource requirements, wherein: include: Determine one or more subtasks included in the offline workflow; For any target subtask among the one or more subtasks, based on the description information of the target subtask, determine a preset number of historical subtasks that are similar to the target subtask, where the description information is used to characterize the attributes of the target subtask and the dependency relationship between the target subtask and other subtasks; based on the actual resource consumption values ​​of the preset number of historical subtasks, determine the resource requirements of the target subtask, and generate and run the target subtask based on the resource requirements of the target subtask.

2. The method according to claim 1, wherein: The determining, based on the description information of the target subtask, a preset number of historical subtasks similar to the target subtask includes: performing vectorization processing on the description information of the target subtask to obtain a description vector of the target subtask; determining a similarity between the description vector of the target subtask and the description vectors of historical subtasks stored in a vector database, and determining the preset number of historical subtasks based on the similarity.

3. The method according to claim 2, wherein: The method further includes: collecting the real resource consumption value of the target subtask at preset time intervals during the execution of the target subtask, and after the target subtask completes execution, storing the maximum value of the description vector of the target subtask and the collected real resource consumption value of the target subtask in the vector database.

4. The method according to any one of claims 1 to 3, wherein: The description information includes multiple items of the following information: the name of the offline workflow, the name of the target subtask, the name of the subtask that the target subtask depends on, the image file name of the target subtask, and the script of the target subtask.

5. The method according to any one of claims 1 to 4, wherein: Determining the resource requirement of the target subtask based on the real resource consumption values ​​of the preset number of historical subtasks includes: processing the real resource consumption values ​​of the preset number of historical subtasks according to a preset rule, and determining the resource requirement of the target subtask based on the processing result.

6. The method according to any one of claims 1 to 4, wherein: Determining the resource requirement of the target subtask based on the actual resource consumption values ​​of the preset number of historical subtasks includes: if the actual resource consumption value of a first historical subtask among the preset number of historical subtasks exceeds a first threshold, determining the resource requirement of the target subtask to be a value greater than the resource requirement of the first historical subtask.

7. The method according to any one of claims 1 to 6, wherein: The method further includes: if the target subtask fails to execute and the reason for the failure is insufficient memory, increasing the memory resource requirement in the resource requirement of the target subtask, rebuilding the target subtask and running it again.

8. The method according to any one of claims 1 to 7, wherein: Before generating and running the target subtask based on the resource requirements of the target subtask, the method further includes: outputting the resource requirements of the target subtask to a user interface and receiving a resource requirement feedback instruction input by a user; and determining the resource requirements of the target subtask or adjusting the resource requirements of the target subtask based on the resource requirement feedback instruction.

9. A resource allocation method, wherein: include: Obtaining a resource requirement of a target subtask of an offline workflow, wherein the offline workflow includes one or more subtasks, the target subtask is any subtask of the offline workflow, and the resource requirement of the target subtask is determined based on actual resource consumption values ​​of a preset number of historical subtasks similar to the target subtask, wherein the preset number of historical subtasks is determined based on description information of the target subtask, where the description information is used to characterize attributes of the target subtask and dependencies between the target subtask and other subtasks; Resources are allocated to the target subtask based on the resource requirements of the target subtask, so as to generate and run the target subtask.

10. An electronic device, wherein: include: memory and processor; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory, and when the computer program is executed, the processor is enabled to execute the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 9.

12. A computer program product, wherein: The method comprises a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images