Resource scheduling method and electronic equipment

Abstract

The invention discloses a resource scheduling method and electronic equipment, and relates to the technical field of resource allocation and scheduling. The resource scheduling method comprises the following steps: in response to a processing model running on a processor, determining model characteristics of the running processing model; the model features at least comprise model categories of the processing models, and the model categories can be used for representing demand features of the processing models for processor resources; and performing resource scheduling of processor resources on the processing model based on the model features, wherein the resource scheduling is used for matching the resources allocated to the processing model with the model features of the processing model.

Description

Technical Field

[0001] This application relates to the field of resource allocation and scheduling technology, and in particular to a resource scheduling method and electronic device. Background Technology

[0002] With the explosion of generative AI, various AI (Artificial Intelligence) models, such as large language models and diffusion models, are widely used in various products.

[0003] In related technologies, processors typically use static or coarse-grained resource allocation strategies to allocate processor resources to AI models running on them, which cannot achieve fine-grained resource scheduling and can easily lead to a decline in the overall performance of the model and a deterioration in energy efficiency. How to solve or improve this problem has become a research hotspot in this field. Summary of the Invention

[0004] Therefore, this application discloses the following technical solution:

[0005] A resource scheduling method, comprising:

[0006] In response to a processing model running on a processor, the model characteristics of the running processing model are determined; the model characteristics include at least the model category of the processing model, which can be used to characterize the processing model's requirements for processor resources;

[0007] Based on the model characteristics, the processor resources of the processing model are scheduled, and the resource scheduling is used to match the resources allocated to the processing model with the model characteristics of the processing model.

[0008] Optionally, in response to a processing model running on the processor, the model characteristics of the running processing model are determined, including:

[0009] In response to the presence of at least two processing models running on the processor, the model category of each processing model is determined.

[0010] The processor resource requirements of each processing model are determined based on its model category.

[0011] Among them, the model characteristics of the processing model also include the processing model's requirements for processor resources.

[0012] Optionally, the processor resource requirements of each processing model can be determined based on its model category, including:

[0013] In response to the model category indicating the existence of a first type of processing model among the processing models, the current processor resource requirements of the first type of processing model are determined based on the model category of the first type of processing model and the current processing stage of the first type of processing model.

[0014] In response to the model category indicating the existence of a second type of processing model among the processing models, the current processor resource requirements of the second type of processing model are determined based on the model category and / or operator features of the second type of processing model.

[0015] Optionally, based on the model category of the first type of processing model and the current processing stage of the first type of processing model, the current processor resource requirements of the first type of processing model are determined, including:

[0016] In response to the fact that the first type of processing model is a language model and is in the pre-filling stage of the inference process, the current demand feature of the first type of processing model for processor resources is determined as the first demand feature;

[0017] In response to the fact that the first type of processing model is a language model and is in the decoding stage of the inference process, the current demand feature of the first type of processing model for processor resources is determined as the second demand feature;

[0018] Wherein, the first demand feature indicates that the resource requirement of the first type of processing model is a high computing power resource requirement; the second demand feature indicates that the resource requirement of the first type of processing model is a high bandwidth resource requirement.

[0019] Optionally, before determining the current processor resource requirements of the first type of processing model based on its model category and current processing stage, the method further includes:

[0020] Determine the current processing stage of the first type of processing model;

[0021] Determining the current processing stage of the first type of processing model includes:

[0022] Obtain the attributes of the input data of the first type of processing model in the current processing stage;

[0023] Based on the attributes of the input data, the current processing stage of the first type of processing model is identified.

[0024] Optionally, the current processor resource requirements of the second type of processing model are determined based on the model category and / or operator characteristics of the second type of processing model, including at least one of the following:

[0025] Based on the model category and target mapping table of the second type of processing model, the current computing power requirement of the second type of processing model for processor resources is determined; the target mapping table includes mapping relationship information between different model categories of the second type of processing model and different computing power requirements;

[0026] Based on the operator characteristics of the operators invoked during the inference process of the second type of processing model, the current computing power requirement of the second type of processing model for processor resources is determined.

[0027] Optionally, before determining the current computing power requirement of the second type of processing model for processor resources based on the operator characteristics of the operators invoked during the inference process of the second type of processing model, the method further includes:

[0028] Determine the operator features of the operators invoked during the inference process of the second type of processing model;

[0029] Among them, determining the operator features of the operators invoked during the inference process of the second type of processing model includes:

[0030] Identify the operator types of each operator included in the second type of processing model to obtain the identification results;

[0031] The second type of processing model is triggered to perform forward propagation based on sample data. During the forward propagation of the second type of processing model, the various types of operators represented by the recognition results are tracked to capture the operator type and / or number of operators called during the forward propagation of the second type of processing model. The operator features include operator type and / or number of operators.

[0032] Optionally, in response to a processing model running on the processor, the model characteristics of the running processing model are determined, including:

[0033] In response to the presence of a processing model running on the processor, a model category of the processing model is determined, and processor resources are scheduled for the processing model based on the model category.

[0034] Optionally, resource scheduling of processor resources for the processing model based on the model category includes:

[0035] In response to the model category characterizing the processing model as a first type of processing model, computing power resources and bandwidth resources that meet the first resource conditions are allocated to the processing model;

[0036] In response to the model category indicating that the processing model is a second type of processing model, computing power resources and bandwidth resources that meet the second resource conditions are allocated to the processing model.

[0037] An electronic device, comprising:

[0038] Model-aware unit;

[0039] A processor is configured to perform the following processing by executing a set of instructions stored in the memory:

[0040] In response to the presence of a processing model running on the processor, the model awareness unit determines the model characteristics of the running processing model; the model characteristics include at least the model category of the processing model, which can be used to characterize the processing model's demand for processor resources;

[0041] Based on the model characteristics, the processor resources of the processing model are scheduled, and the resource scheduling is used to match the resources allocated to the processing model with the model characteristics of the processing model.

[0042] A storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement any of the resource scheduling methods provided above. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating one of the resource scheduling methods provided in this application;

[0045] Figure 2 This is another flowchart illustrating the resource scheduling method provided in this application;

[0046] Figure 3 This is a schematic diagram of the LLM operation at different processing stages provided in this application;

[0047] Figure 4 This is a flowchart illustrating the operator features used by the operators in the model defined in this application.

[0048] Figure 5 This is a schematic diagram of the customized model operator sensing unit for the model kernel layer provided in this application;

[0049] Figure 6 This is another flowchart illustrating the resource scheduling method provided in this application;

[0050] Figure 7 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a resource scheduling method and an electronic device. The method can be applied to electronic devices in a variety of general or special computing environments or configurations, such as personal computers, server computers, smart terminals, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0053] See Figure 1 The flowchart shown illustrates the resource scheduling method provided in this application embodiment, which may include the following steps 101 to 102, which are described in detail below.

[0054] Step 101: In response to the processing model running on the processor, determine the model characteristics of the running processing model.

[0055] The processing model may include, but is not limited to, one or more of various AI models such as CV (Computer Vision) models, LLM (Large Language Model) models, segmentation models, speech recognition models, and image generation models.

[0056] The processor can be, but is not limited to, GPUs (Graphics Processing Units), NPUs (Neural-network Processing Units), or other processors that can be used to run AI models.

[0057] In this embodiment of the application, when a processing model is running on the processor, such as when one or more AI models are running, the model characteristics of the running processing model are determined.

[0058] The model features include at least the model category of the processing model, which can be used to characterize the processing model's requirements for processor resources.

[0059] Optionally, the model category can be a model category obtained by classifying the processing model from the dimension of resource requirements, such as a computing power model category that represents a model with high computing power resource requirements, a bandwidth model category that represents a model with high bandwidth resource requirements, etc.; or, the model category can also be a category obtained by classifying the model from the dimension of non-resource requirements such as model function and attributes, such as CV model, LLM model, speech recognition model, image generation model, etc.

[0060] In cases where model categories are divided based on non-resource requirement dimensions such as model function and attributes, a mapping relationship between different model categories and their respective processor resource requirements can be pre-defined based on the actual processor resource requirements of different model categories. Alternatively, the actual operation of the processing model can be dynamically determined, such as the processing stage of the processing model, the number and / or type of operators called, etc., so that the model category of the processing model can characterize the processor resource requirement of the processing model based on the mapping relationship or the actual operation of the processing model.

[0061] The processing model's requirements for processor resources may include, but are not limited to, the requirements for processor computing power, bandwidth, and / or cache.

[0062] Step 102: Based on the model characteristics, perform resource scheduling of processor resources on the processing model. The resource scheduling is used to match the resources allocated to the processing model with the model characteristics of the processing model.

[0063] In the process of scheduling processor resources for the processing model based on the model features, optionally, the processing model's demand for processor resources can be allocated and scheduled based on the model category characteristics in the model features, so that the resources allocated to the processing model match the processing model's demand for processor resources.

[0064] For example, when the processing model is a computing power-type model, high computing power resources and low bandwidth resources are allocated and scheduled to the processing model; when the processing model is a bandwidth-type model, high bandwidth resources and low computing power resources are allocated and scheduled to the processing model. Alternatively, for model types that are not based on resource requirements, such as CV models or LLM models, processor resources (such as computing power and bandwidth) of the corresponding resource type and scale are adaptively allocated based on their actual operating conditions, such as the current processing stage, the number and / or type of operators called.

[0065] In summary, the resource scheduling method provided in this application, when a processing model is running on a processor, determines the model characteristics of the running processing model, ensuring that the determined model characteristics include at least model categories that can characterize the processing model's demand for processor resources. Then, based on these model characteristics, the method performs processor resource scheduling on the processing model, ensuring that the resources allocated to the processing model match its model characteristics. Therefore, this application achieves refined processor resource allocation and scheduling based on model characteristics (model features), effectively meeting the resource requirements of the processing model running on the processor. This avoids overall performance degradation and energy efficiency deterioration caused by unreasonable resource allocation and scheduling, thereby guaranteeing the performance of the processing model.

[0066] In an alternative embodiment, see Figure 2 The schematic diagram of the resource scheduling method shown in this application illustrates that step 101 in the method provided in this application, which involves determining the model characteristics of the running processing model in response to the processor running a processing model, can be implemented as follows: steps 201-202:

[0067] Step 201: In response to the fact that at least two processing models are running on the processor, determine the model category of each processing model.

[0068] When at least two processing models are running on a processor, the model category of each processing model is identified. Specifically, the model category of each processing model can be determined by reading the attribute information of the processing model and / or by inductively analyzing the resource requirements during the operation of different models.

[0069] It should be noted that the resource demand situation here refers to relatively coarse overall resource demand information for processing the model, such as whether the model mainly has a high demand for computing power resources or mainly has a high demand for bandwidth resources, etc.

[0070] The determined processing model category may include, but is not limited to, model categories divided from the dimensions of resource requirements and / or functions and attributes, such as computing power / bandwidth models, CV / LLM models, etc.

[0071] Step 202: Determine the processor resource requirements of each processing model based on its model category.

[0072] This step can be further implemented as follows: steps “1-1”-“1-2”.

[0073] 1-1: In response to the model category characterization that there is a first type of processing model among the processing models, based on the model category of the first type of processing model and the current processing stage of the first type of processing model, determine the current processor resource requirements of the first type of processing model.

[0074] The first type of processing model refers to a processing model that has highly differentiated requirements for processor resources at different processing stages during model operation. These highly differentiated requirements may, but are not limited to, mean that the types of processor resources required by the processing model differ at different processing stages, and / or that the difference in resource requirements for the same type of resource reaches a threshold.

[0075] The first type of processing model can be, but is not limited to, LLM-type models. In the prefill stage of the inference process, LLM-type models mainly involve matrix-matrix multiplication (GEMM), which requires high computational power but relatively low bandwidth. In the decoding stage, the operations mainly involve matrix-vector multiplication (GEMV), which requires high bandwidth but low computational power.

[0076] Based on the aforementioned characteristics of the first type of processing model, the current processing stage of the first type of processing model can be determined first. Then, by combining the model category of the first type of processing model and the current processing stage of the first type of processing model, the current processor resource requirements of the first type of processing model can be determined.

[0077] Optionally, determining the current processing stage of the first type of processing model can be further implemented by: obtaining the attributes of the input data of the first type of processing model in the current processing stage, and identifying the current processing stage of the first type of processing model based on the attributes of the input data.

[0078] The properties of the obtained input data may include, but are not limited to, the tensor shape of the input data (the shape of the input tensor), which represents the size of the input data in each dimension.

[0079] The first type of processing model has different tensor shapes for the input data at different processing stages. For example, see LLM. Figure 3The diagram illustrates the operations at different processing stages. In the pre-filling stage, the operations mainly involve matrix-matrix multiplication and addition (GEMM) operations. Correspondingly, the tensor shape of the input data represents the size of the input data in each dimension for the GEMM operation. In the decoding stage, the operations mainly involve matrix-vector multiplication and addition (GEMV) operations. The tensor shape of the input data also represents the size of the input data in each dimension for the GEMV operation. The input data differs between the two stages, and therefore, the tensor shapes (i.e., the sizes of the input data in each dimension) also differ. Consequently, the current processing stage of a first-type processing model, such as LLM, can be identified by detecting the tensor shape of the input data corresponding to the current processing stage.

[0080] Specifically, taking LLM as an example, the stage of LLM can be determined by checking whether the second dimension of the shape of the input tensor is 1. If the second dimension of the shape of the input tensor is 1, the current operation is GEMV, which indicates that the LLM is in the decoding stage. If the second dimension of the shape of the input tensor is greater than 1, the current operation is GEMM, which indicates that the LLM is in the pre-filling stage.

[0081] Specifically, if the first type of processing model is a language model (such as an LLM-type model) and is in the pre-filling stage of the inference process, it has high computing power requirements but relatively low bandwidth requirements. Therefore, the current processor resource requirements of the first type of processing model can be identified as the first requirement characteristic representing high computing power resource requirements. If the first type of processing model is a language model and is in the decoding stage of the inference process, it has high bandwidth requirements but low computing power requirements. Therefore, the current processor resource requirements of the first type of processing model can be identified as the second requirement characteristic representing high bandwidth resource requirements.

[0082] 1-2: In response to the model category characterizing the existence of a second type of processing model among the processing models, the current processor resource requirements of the second type of processing model are determined based on the model category and / or operator features of the second type of processing model.

[0083] The second type of processing model is a computing power-type model, which has high computing power requirements and relatively low bandwidth requirements. The second type of processing model can be, but is not limited to, CV-type models, such as face detection models, face recognition models, human body detection models, human pose detection models, etc.

[0084] Optionally, determining the current processor resource requirements of the second-type processing model based on its model category and / or operator characteristics can be further implemented as including at least one of the following implementation methods "2-1"-"2-2":

[0085] 2-1: Based on the model category and target mapping table of the second type of processing model, determine the current computing power requirement of the second type of processing model for processor resources.

[0086] The target mapping table includes information on the mapping relationship between different model categories of the second type of processing model and different computing power requirements.

[0087] In the target mapping table, the model category of the second type of processing model can be characterized by, but is not limited to, information such as the function / task type and / or model name corresponding to the second processing model. See the example in Table 1, which provides mapping information between different CV class models and their corresponding computing power requirements.

[0088] Table 1

[0089] In this implementation method 2-1, the target mapping table can be looked up based on the model category of the second type of processing model (such as the face detection model YOLOv5-Face, etc.) to obtain the computing power information that matches its model category, which can be used as the computing power requirement feature of the second type of processing model.

[0090] 2-2: Based on the operator characteristics of the operators called during the inference process of the second type of processing model, determine the current computing power requirements of the second type of processing model for processor resources.

[0091] Optionally, the operator features of the operators called during the inference process of the second type of processing model may include, but are not limited to, the operator type and / or the number of operators called by the operators at the current processing progress of the second type of processing model during the inference process.

[0092] During model operation, the types and / or number of operators invoked at different processing stages are usually different. This will lead to different types and / or quantities of processor resources required by the model at different processing stages. Based on this, this implementation can first determine the operator characteristics such as the operator type and / or quantity of operators invoked by the second type of processing model at the current processing stage, and then determine the computing power requirements of the second type of processing model for processor resources based on the operator type and / or quantity of operators invoked by the second type of processing model at the current stage.

[0093] In this embodiment, the model characteristics of each processing model running on the processor include the model category and the model's processor resource requirements (such as computing power, bandwidth, and other resource requirements) determined in steps 201 and 202, respectively. For example, this may specifically include the model category of the first type of processing model and the current processor resource requirements of the first type of processing model determined in step 1-1, as well as the model category of the second type of processing model and the current processor resource requirements of the second type of processing model determined in step 1-2.

[0094] After obtaining the model characteristics of each processing model running on the processor, the processor resources can be allocated and scheduled for each processing model by comprehensively considering the model characteristics of each processing model, so that the resources allocated to each processing model match the model characteristics of each processing model.

[0095] In addition, resource allocation can also take into account the computational priority of different processing models when scheduling resources for each processing model.

[0096] The following example illustrates the resource scheduling method of this application, using the simultaneous running of LLM and CV models on a processor.

[0097] When both LLM and CV models are running simultaneously on processors such as NPUs, the processor can determine the resource requirements of the LLM model based on its model category and current inference stage (such as pre-filling or decoding). It can also determine the resource requirements of the CV model based on its model category and the operator type and number of operators called at the current progress. This allows for resource allocation and scheduling of the two models by combining their model categories and resource requirements, or by combining computational priorities.

[0098] If the LLM is currently running in the prefill stage, which mainly involves matrix-matrix multiplication and addition (GEMM) operations, it has high computational power requirements but relatively low bandwidth requirements. The priority of the LLM and CV-type models can be combined to allocate most of the processor's computational power to the LLM model, enabling faster computation and reducing the user's first-word waiting time. If the LLM is currently running in the decoding stage, which mainly involves matrix-vector multiplication and addition (GEMV) operations, it has high bandwidth requirements but low computational power requirements. A large amount of computational power can be allocated to the CV-type model while a large amount of bandwidth can be allocated to the LLM. If the processor's internal cache space is sufficient, all bandwidth can also be allocated to the LLM. This allows the LLM to share all memory bandwidth to achieve maximum decoding speed without compromising the performance of the CV-type model. This allows both the LLM and CV-type models to run at their maximum computational speed, maintaining optimal computational speed for both models and improving the user experience.

[0099] When different models run in parallel on the same processor, severe resource contention often occurs. In related technologies, processors typically use static or coarse-grained resource allocation strategies to allocate processor resources to the processing models running on them, which cannot achieve fine-grained resource scheduling and can easily lead to a serious decline in the overall performance of the model and a deterioration in energy efficiency.

[0100] This embodiment determines the model characteristics of each processing model running on the processor, and performs processor resource allocation and scheduling for each processing model based on the model characteristics of each processing model. It ensures that the resources allocated to each processing model match the model characteristics of each processing model, thereby achieving fine-grained resource scheduling for each processing model. Based on the model characteristics of each processing model, it can effectively meet or as much as possible meet the resource requirements of each processing model during operation, thereby avoiding model performance degradation and energy efficiency ratio deterioration, and ensuring the model performance of each processing model on the processor.

[0101] In an alternative embodiment, see Figure 4 The operator features of the operators called during the inference process of the second type of processing model are determined, which can be implemented, but is not limited to, the following processing steps 401-402:

[0102] Step 401: Identify the operator types of each operator included in the second type of processing model and obtain the identification results.

[0103] Step 401 can be further implemented as at least one of steps “3-1”-“3-2” as follows:

[0104] 3-1: Basic review of model operators.

[0105] In the inference engine architecture that processes the model, the Kernel layer is the core component that performs low-level mathematical operations. It interacts directly with the hardware and is responsible for implementing computationally intensive operations in the model. Its specific information and functions mainly revolve around the implementation and optimization of high-performance operators.

[0106] The kernel layer comprises the set of code responsible for performing low-level computations within the model, containing operator information required for the operations of each network layer. As a core component of the inference engine, the kernel layer performs low-level mathematical operations on hardware, which are implemented through various operators.

[0107] Operators define specific computational logic, such as convolution, matrix multiplication, and activation functions, while kernels are used to implement these operators on specific hardware such as GPUs and NPUs.

[0108] Based on this, a preliminary identification of the required operators for each network layer of the model can be achieved by performing a basic examination of the model's kernel layer. Optionally, this basic examination can refer to extracting the model's structural framework from the kernel layer. The extracted structural framework contains the operator information required by each network layer. Therefore, the required operators for each network layer can be preliminarily identified based on the extracted structural framework, and preliminary identification results can be output.

[0109] The network layers of the model may include, but are not limited to, convolutional layers, fully connected layers, attention layers, activation layers, pooling layers, and / or normalization layers, etc.

[0110] Optionally, the preliminary identification results output may include class names of each network layer of the model. Each class name is used to represent the corresponding operator type. The following is an example of multiple class names and the operator types they represent in the identification results:

[0111] nn.Conv2d: Two-dimensional convolution operator;

[0112] nn.Linear: A linear (fully connected) operator;

[0113] nn.ReLU: ReLU activation operator;

[0114] nn.MaxPool2d: Max pooling operator;

[0115] nn.BatchNorm2d: Batch normalization operator.

[0116] 3-2: In-depth examination of model operators.

[0117] For complex models (such as AI models containing Sequential or custom modules), each network layer may contain multiple basic operators, such as the multiple basic operators contained within a custom module of a network layer (such as BasicBlock). These operators cannot be identified from the kernel layer. For these operators, this embodiment further identifies them by examining the network layer definition code. Based on this, each network layer of the complex model can be traversed layer by layer to deeply examine and identify each basic operator by traversing the sub-modules of each network layer.

[0118] Step 402: Trigger the forward propagation of the second type of processing model based on the sample data. During the forward propagation of the second type of processing model, track the various types of operators represented by the recognition results to capture the operator types and number of operators called during the forward propagation of the second type of processing model.

[0119] Based on step 401, the second type of processing model can be further triggered to perform forward propagation based on sample data. Based on the actual calculation process of its forward propagation, the actual calling of each type of operator represented by the recognition result can be tracked at each processing stage (such as convolution, attention operation, normalization, etc.) during the forward propagation of the model. This captures information such as the operator type and number of operators called at each processing stage during the forward propagation of the second type of processing model, and obtains the operator features of the operators called at each stage during the inference process of the second type of processing model.

[0120] In practical use, optionally, random inputs of the corresponding size can be defined as sample data according to the input data dimension required by the second type of processing model to trigger the second type of processing model to perform forward propagation, so as to capture operator calls based on the actual calculations in its forward propagation process.

[0121] In practice, this is optional; see [link / reference]. Figure 5 A custom model operator awareness unit can be customized for the model kernel layer. This unit identifies the type and quantity of operators invoked by the model at each stage of the process described above. For example, within the processor's scheduler, a corresponding model operator awareness unit can be set up for each running processing model. When operator feature identification is required for each processing model, the corresponding model operator awareness unit can perceive the operator type, quantity, and other operator characteristics of each processing model running on the processor. Then, based on the type and quantity of operators scheduled by the model at the current stage, the processor's internal scheduler dynamically allocates processor resources such as computing power, bandwidth, cache, and computational priority. This effectively improves the utilization of computing power and bandwidth of processors such as the NPU, thereby enhancing the inference performance of different types of models.

[0122] This embodiment can precisely identify the operator types and quantities of operators actually called at different stages in the reasoning process of the second type of processing model. This facilitates the dynamic determination of resource requirements at different stages in the reasoning process of the second type of processing model based on the corresponding operator characteristics, so as to dynamically and adaptively allocate and schedule resources for the second type of processing model at different processing stages.

[0123] In an alternative embodiment, see Figure 6 The schematic diagram of the resource scheduling method shown in this application illustrates that step 101 in the method provided in this application, which involves determining the model characteristics of the running processing model in response to the processor running a processing model, can also be implemented as follows:

[0124] Step 601: In response to the presence of a processing model running on the processor, determine the model category of the processing model, and perform resource scheduling of processor resources for the processing model based on the model category.

[0125] The determined processing model category may include, but is not limited to, model categories divided from the dimensions of resource requirements and / or functions and attributes, such as computing power / bandwidth models, CV / LLM models, etc.

[0126] Based on the determination of the model category of a processing model, processor resources can be scheduled for that processing model according to the model category.

[0127] Specifically, in response to the model category indicating that the processing model is a first type of processing model, computing power resources and bandwidth resources that meet the first resource conditions can be allocated to the processing model; in response to the model category indicating that the processing model is a second type of processing model, computing power resources and bandwidth resources that meet the second resource conditions can be allocated to the processing model.

[0128] The first computing power condition can be used to characterize low computing power resources and high bandwidth resources. The second computing power condition can be used to characterize high computing power resources and low bandwidth resources.

[0129] Among them, the level of computing power / bandwidth resources is a relative and quantifiable concept. In practical applications, the level of computing power / bandwidth resources can be characterized by the proportion or amount of resources set / allocated, but is not limited to.

[0130] As mentioned above, the first type of processing model can be LLM. When running a single model on the processing model, the resource requirements of the single model can be determined in a relatively coarse-grained manner. For example, the LLM can be classified as a bandwidth-type model. Based on this model category (bandwidth-type model), a certain amount of processor computing power resources can be allocated to the model according to its computing power requirements. At the same time, unnecessary computing power units can be shut down, and all the processor's bandwidth can be allocated to the model to achieve the maximum model computing performance.

[0131] The second type of processing model can be a CV-type model, and can be judged as a computing power-type model as a whole. When the processor runs this single model, optionally, all computing units of the processor can be allocated to this model for calculation to achieve the effect of fast calculation.

[0132] When running a single model on a processor, this embodiment determines the resource requirements of the single model in a relatively coarse-grained manner, and allocates and schedules resources for the model based on this, which simplifies the processing logic of resource allocation and scheduling and improves resource scheduling efficiency.

[0133] It is easy to understand that, for the case of running a single model on a processor, the implementation method is not limited to the above. In practice, more refined information such as the type and number of operators scheduled at different processing stages can be used to identify the single model. For example, identifying whether the LLM is in the pre-filling stage or the decoding stage, or identifying the type and number of operators scheduled at different processing stages for CV-type models, can be used to achieve more refined and rational resource allocation and scheduling for the single model. This can ensure high computational performance of the single model while avoiding resource waste.

[0134] It should be noted that the LLM or CV models provided in this application embodiment are merely examples and do not constitute a limitation on the models running on the processor. In practical applications, other types of models, such as segmentation models, speech recognition models, and image generation models, can also run on the processor. For these models, processor resources can be dynamically and precisely allocated and scheduled based on their actual resource requirements using the method of this application. For example, image generation models and speech recognition models are both computationally intensive models and can be classified into the category of computationally intensive models. The required computational resources can be adaptively allocated to them based on the resource requirements of computationally intensive models (such as the computational requirements corresponding to the types and quantities of operators called at different progress stages).

[0135] In practical applications, the models running on the processor may be composite models, meaning that the types and / or quantities of resource requirements will change significantly at different stages of the model's operation. For example, the LLM and other first-class models mentioned above are essentially composite models. In this case, preferably, the resource requirements at different stages of the model's operation (such as the pre-filling stage and decoding stage of LLM) can be determined separately to perform dynamic adaptive resource allocation and scheduling, so as to ensure high computational performance of the model while avoiding resource waste.

[0136] This application also discloses an electronic device, the composition and structure of which are as follows: Figure 7 As shown, it includes at least:

[0137] Memory 10 is used to store the computer instruction set;

[0138] Computer instruction sets can be implemented in the form of computer programs.

[0139] Processor 20 is configured to implement the resource scheduling method provided in any of the above method embodiments by executing a set of computer instructions in the memory.

[0140] The processor 20 can be a central processing unit (CPU), a graphics processor, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices.

[0141] Optionally, the electronic device may also include a display device for displaying information.

[0142] Optionally, electronic devices may also include storage resources such as memory and cache.

[0143] Optionally, the electronic device may also include an image acquisition device.

[0144] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0145] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0146] This application also discloses a storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement the resource scheduling method provided in any of the above method embodiments.

[0147] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0148] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0149] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0150] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A resource scheduling method, comprising: In response to a processing model running on a processor, the model characteristics of the running processing model are determined; the model characteristics include at least the model category of the processing model, which can be used to characterize the processing model's requirements for processor resources; Based on the model characteristics, the processor resources of the processing model are scheduled, and the resource scheduling is used to match the resources allocated to the processing model with the model characteristics of the processing model.

2. The resource scheduling method according to claim 1, in response to a processing model running on the processor, determining the model characteristics of the running processing model, including: In response to the presence of at least two processing models running on the processor, the model category of each processing model is determined. The processor resource requirements of each processing model are determined based on its model category. Among them, the model characteristics of the processing model also include the processing model's requirements for processor resources.

3. The resource scheduling method according to claim 2, wherein the processor resource requirements of each processing model are determined based on the model category of each processing model, including: In response to the model category indicating the existence of a first type of processing model among the processing models, the current processor resource requirements of the first type of processing model are determined based on the model category of the first type of processing model and the current processing stage of the first type of processing model. In response to the model category indicating the existence of a second type of processing model among the processing models, the current processor resource requirements of the second type of processing model are determined based on the model category and / or operator features of the second type of processing model.

4. The resource scheduling method according to claim 3, based on the model category of the first type of processing model and the current processing stage of the first type of processing model, determines the current processor resource requirements of the first type of processing model, including: In response to the fact that the first type of processing model is a language model and is in the pre-filling stage of the inference process, the current demand feature of the first type of processing model for processor resources is determined as the first demand feature; In response to the fact that the first type of processing model is a language model and is in the decoding stage of the inference process, the current demand feature of the first type of processing model for processor resources is determined as the second demand feature; Wherein, the first demand feature indicates that the resource requirement of the first type of processing model is a high computing power resource requirement; the second demand feature indicates that the resource requirement of the first type of processing model is a high bandwidth resource requirement.

5. The resource scheduling method according to claim 3 or 4, before determining the current processor resource requirements of the first type of processing model based on the model category of the first type of processing model and the current processing stage of the first type of processing model, further includes: Determine the current processing stage of the first type of processing model; Determining the current processing stage of the first type of processing model includes: Obtain the attributes of the input data of the first type of processing model in the current processing stage; Based on the attributes of the input data, the current processing stage of the first type of processing model is identified.

6. The resource scheduling method according to claim 3, wherein the current processor resource requirement characteristics of the second type of processing model are determined based on the model category and / or operator characteristics of the second type of processing model, including at least one of the following: Based on the model category and target mapping table of the second type of processing model, the current computing power requirement of the second type of processing model for processor resources is determined; the target mapping table includes mapping relationship information between different model categories of the second type of processing model and different computing power requirements; Based on the operator characteristics of the operators invoked during the inference process of the second type of processing model, the current computing power requirement of the second type of processing model for processor resources is determined.

7. The resource scheduling method according to claim 6, before determining the current computing power requirement of the second type of processing model for processor resources based on the operator characteristics of the operators invoked during the inference process of the second type of processing model, further includes: Determine the operator features of the operators invoked during the inference process of the second type of processing model; Among them, determining the operator features of the operators invoked during the inference process of the second type of processing model includes: Identify the operator types of each operator included in the second type of processing model to obtain the identification results; The second type of processing model is triggered to perform forward propagation based on sample data. During the forward propagation of the second type of processing model, the various types of operators represented by the recognition results are tracked to capture the operator type and / or number of operators called during the forward propagation of the second type of processing model. The operator features include operator type and / or number of operators.

8. The resource scheduling method according to claim 1, in response to a processing model running on the processor, determining the model characteristics of the running processing model, including: In response to the presence of a processing model running on the processor, a model category of the processing model is determined, and processor resources are scheduled for the processing model based on the model category.

9. The resource scheduling method according to claim 8, wherein resource scheduling of processor resources is performed on the processing model based on the model category, comprising: In response to the model category characterizing the processing model as a first type of processing model, computing power resources and bandwidth resources that meet the first resource conditions are allocated to the processing model; In response to the model category indicating that the processing model is a second type of processing model, computing power resources and bandwidth resources that meet the second resource conditions are allocated to the processing model.

10. An electronic device, comprising: Model-aware unit; A processor is configured to perform the following processing by executing a set of instructions stored in the memory: In response to the presence of a processing model running on the processor, the model awareness unit determines the model characteristics of the running processing model; the model characteristics include at least the model category of the processing model, which can be used to characterize the processing model's demand for processor resources; Based on the model characteristics, the processor resources of the processing model are scheduled, and the resource scheduling is used to match the resources allocated to the processing model with the model characteristics of the processing model.

Images