Method and apparatus for scheduling and reasoning of large model parameters, and electronic device

Abstract

The application provides a large model parameter scheduling method, an inference method, a device and electronic equipment. The method comprises: obtaining input features of a current layer transformer module; the input features are features output by a previous layer transformer module; analyzing the input features by a parameter prediction model corresponding to the current layer transformer module to obtain a target expert model required for inference of a next layer transformer module; if it is determined that target model parameters need to be scheduled based on the target expert model, generating a parameter scheduling strategy; and scheduling the target model parameters of the target expert model from CPU memory to GPU memory. The application obtains the target expert model required for inference of the next layer transformer module through the parameter prediction model, and schedules the target model parameters from the CPU to the GPU, thereby reducing the occupation of the GPU memory.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a scheduling method, inference method, apparatus, and electronic device for large model parameters. Background Technology

[0002] With the rapid iteration and updates of large language models, sparse hybrid expert models (MoE) have attracted much attention as a novel model architecture. Introducing sparse hybrid expert models into large models aims to improve the model's expressive power and performance while maintaining computational efficiency.

[0003] In the process of large model inference, the model parameters of the expert model are written into the GPU and then the GPU is used for inference. Since the number of expert models activated in each inference module is small, only a very small portion of the model parameters written into the GPU are required for inference, resulting in a waste of GPU memory resources. Summary of the Invention

[0004] The purpose of this application is to provide a scheduling method, inference method, apparatus, and electronic device for large model parameters, so as to reduce the waste of GPU memory resources.

[0005] In a first aspect, embodiments of this application provide a method for scheduling parameters of a large model, the large model including multiple transformer modules, and a parameter prediction model set for one transformer module. The method includes:

[0006] Obtain the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module.

[0007] By analyzing the input features through the parameter prediction model corresponding to the current transformer module, the target expert model required for inference in the next transformer module is obtained.

[0008] Determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model.

[0009] If necessary, generate a parameter scheduling strategy;

[0010] According to the parameter scheduling strategy, the target model parameters of the target expert model are scheduled from the CPU memory to the GPU memory.

[0011] This application embodiment uses the parameter prediction model corresponding to each transformer module to predict the input features, obtains the target expert model required for the inference of the next transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the GPU memory usage.

[0012] In one possible implementation of the first aspect, determining whether the target model parameters of the target expert model need to be scheduled based on the target expert model includes:

[0013] Determine whether the target model parameters of the target expert model are stored in the GPU memory;

[0014] If the target model parameters are not stored in the GPU memory, then scheduling is required; otherwise, scheduling is not required.

[0015] This application embodiment determines whether target model parameters need to be scheduled by judging whether they are stored in GPU memory, thereby reducing unnecessary scheduling operations and reducing the overhead and time cost of data migration.

[0016] In one possible implementation of the first aspect, the input features are analyzed using the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference in the next layer transformer module, including:

[0017] The input features are fed into the parameter prediction model of the current layer transformer module to obtain the probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to characterize the probability that the expert model is the expert model required for inference in the next layer transformer module;

[0018] Based on probability values, a target expert model is selected from multiple expert models according to the expert model selection strategy.

[0019] In one possible implementation of the first aspect, the expert model selection strategy includes:

[0020] The number of target expert models selected from multiple expert models is greater than a preset number; and / or,

[0021] Starting with the highest probability value, obtain expert models whose sum of probability values ​​is greater than a preset threshold.

[0022] This application embodiment selects a target expert model from multiple expert models through an expert model selection strategy. The number of selected target expert models changes dynamically according to the probability value of each expert model, which can improve the accuracy of expert model parameter scheduling.

[0023] In one possible implementation of the first aspect, the parameter scheduling strategy is generated, including:

[0024] The target model parameters, the model parameters already stored in the GPU memory, and the environmental features are input into the parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model. Among them, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM utilization rate, dynamic random access memory (DRAM) capacity, DRAM utilization rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization rate, and GPU utilization rate.

[0025] The embodiments of this application, by taking into account the current environmental characteristics, can allocate storage and computing resources more effectively, reducing resource waste; and can be applied to heterogeneous systems with different types of memory and processors to make full use of the overall system performance.

[0026] In one possible implementation of the first aspect, the parameter scheduling model is obtained by training based on a reinforcement learning algorithm.

[0027] In this embodiment, reinforcement learning algorithms enable models to self-adjust and optimize based on feedback from the environment. Through continuous learning, they can consider the occupancy and performance characteristics of various resources, thereby making more reasonable resource allocation strategies.

[0028] In one possible implementation of the first aspect, before determining whether the target model parameters of the target expert model need to be scheduled based on the target expert model, the method further includes:

[0029] By analyzing the input features using the parameter prediction model corresponding to the current layer transformer module, the target expert model required for the current layer transformer module to infer the next token is obtained.

[0030] The parameter prediction model in this embodiment can not only predict the target expert model required for the next layer transformer module to infer, but also predict the target expert model required for the current layer transformer module to infer the next token. Therefore, the target expert model required for the transformer module to infer a token is predicted based on the parameter prediction model of the current layer and the parameter prediction model of the previous layer transformer module, thereby improving the accuracy of the prediction.

[0031] In one possible implementation of the first aspect, before scheduling the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy, the method further includes:

[0032] If the GPU memory is insufficient, some model parameters are deleted from the GPU memory based on the deletion rules.

[0033] This application embodiment maintains the model parameters in GPU memory through deletion rules, so that the model parameters of the thermal expert model are always stored in GPU memory, reducing the number of model parameter scheduling times.

[0034] In one possible implementation of the first aspect, the method further includes:

[0035] The cache prediction model is used to predict the target key-value cache data required for the large model to infer subsequent tokens; subsequent tokens refer to tokens that the large model has not yet inferred.

[0036] Determine whether the target key-value cache data needs to be scheduled;

[0037] If scheduling is required, a cache scheduling strategy will be generated;

[0038] According to the cache scheduling policy, the target key-value cache data is obtained from the CPU memory and stored in the GPU memory. The target key-value cache data is used to enable the large model to infer subsequent tokens. The CPU memory is used to store the full amount of key-value cache data. During the inference process, the large model reads the required key-value cache data from the GPU memory.

[0039] This application embodiment predicts the target key-value cache data required for subsequent token inference by a large model using a predictive model, and performs operations such as scheduling judgment, scheduling strategy generation, and data migration on the target key-value cache data, reducing frequent access to the first storage space. Furthermore, only the required target key-value cache data is transmitted each time, rather than the full amount of key-value cache data, which greatly reduces the bandwidth resource occupation caused by transmitting key-value cache data.

[0040] In one possible implementation of the first aspect, the target key-value cache data required for large model inference of subsequent tokens is predicted using a cache prediction model, including:

[0041] Input the already inferred tokens, the current token, and the location information of the current token into the cache prediction model to obtain the target key-value cache data required for subsequent token inference from the cache prediction model; where the current token refers to the token that the large model is inferring.

[0042] This application embodiment utilizes a predictive model to analyze the reasoned tokens, the current token, and the location information of the current token, thereby capturing contextual information and more accurately predicting the target key-value cache data required for subsequent tokens.

[0043] In one possible implementation of the first aspect, the generation of a cache scheduling strategy includes:

[0044] The target key-value cache data, the key-value cache data already stored in GPU memory, and environmental features are input into the cache scheduling model to obtain the cache scheduling policy output by the cache scheduling model. Among them, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM utilization rate, dynamic random access memory (DRAM) capacity, DRAM utilization rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization rate, and GPU utilization rate.

[0045] In this embodiment, the scheduling model can take into account the current environmental characteristics, thereby allocating storage and computing resources more effectively and reducing resource waste; and it can be applied to heterogeneous systems with different types of memory and processors to make full use of the overall system performance.

[0046] Secondly, embodiments of this application provide a large model inference method. The large model includes multiple transformer modules, each transformer module having a parameter prediction model, including:

[0047] Input the reasoned tokens into the large model; the large model consists of multiple transformer modules, and each transformer module contains multiple expert models.

[0048] The active expert model is determined from multiple expert models by using the input features of each transformer module.

[0049] For cases where the target model parameters stored in GPU memory contain the activation model parameters corresponding to all activation expert models, inference is performed on the input features based on the activation model parameters corresponding to the activation expert models to obtain the output features;

[0050] The target model parameters are obtained using the method described in the first aspect.

[0051] This application embodiment uses the parameter prediction model corresponding to each transformer module to predict the input features, obtains the target expert model required for the inference of the next transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the GPU memory usage.

[0052] In one possible implementation of the first aspect, the method further includes:

[0053] For cases where the target model parameters stored in GPU memory contain activation model parameters corresponding to some activation expert models, inference is performed on the input features based on the partial activation model parameters stored in GPU memory and other activation model parameters stored in CPU memory.

[0054] The CPU memory stores the model parameters corresponding to all expert models.

[0055] In this embodiment, the target expert model predicted by the parameter prediction model may be inaccurate. That is, the GPU may not contain all the model parameters of the activated expert model required for inference of a certain transformer layer containing a large model. In this case, the GPU and CPU can be connected to perform inference together, without having to schedule the remaining activated expert model parameters from the CPU to the GPU again, thus improving the model inference efficiency.

[0056] In one possible implementation of the first aspect, the method further includes:

[0057] The target key-value cache data of the token to be inferred is obtained through a large model; wherein, the target key-value cache data is obtained by the scheduling method of the large model parameters in the first aspect;

[0058] Accordingly, inference is performed on the input features based on the activation model parameters corresponding to the activation expert model, including:

[0059] Inference is performed on the input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

[0060] This application embodiment performs inference on input features based on the model parameters corresponding to the activated expert model and the target key-value cache data. The model parameters corresponding to the activated expert model and the target key-value cache data are obtained from targeted scheduling of the CPU, which improves inference efficiency while reducing frequent access to the first storage space.

[0061] In one possible implementation of the first aspect, the inference process of the large model and the scheduling process of the target model parameters and the scheduling process of the target key-value cache data are executed asynchronously.

[0062] This application embodiment executes the inference of a large model and the scheduling of target model parameters and target key-value cache data asynchronously. During the inference process of the current layer transformer module, the target expert model required for the inference of the next layer transformer module is predicted by the parameter prediction model of the current layer, and the corresponding model parameters are scheduled. The target key-value cache data required for subsequent tokens is predicted and scheduled. There is no need to temporarily schedule the model parameters of the target expert model and the target key-value cache data, which improves the inference efficiency of the large model.

[0063] Thirdly, embodiments of this application provide a scheduling device for large model parameters. The large model includes multiple transformer modules and a parameter prediction model corresponding to each transformer module. The device includes:

[0064] The feature acquisition module is used to acquire the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module.

[0065] The expert model prediction module is used to analyze the input features through the parameter prediction model corresponding to the current layer transformer module, and obtain the target expert model required for the inference of the next layer transformer module.

[0066] The decision module is used to determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model.

[0067] The scheduling strategy generation module is used to generate parameter scheduling strategies when it is necessary to schedule the parameters of the target model.

[0068] The scheduling module is used to schedule the target model parameters of the target expert model from CPU memory to GPU memory according to the parameter scheduling strategy.

[0069] Fourthly, embodiments of this application provide a large model inference device, wherein the large model includes multiple transformer modules, and each transformer module includes multiple expert models;

[0070] The expert model determination module is used to determine the active expert model from multiple expert models using the input features of each transformer module.

[0071] The inference module is used to infer the input features based on the activation model parameters corresponding to all activation expert models when the target model parameters stored in GPU memory contain the activation model parameters corresponding to all activation expert models, and obtain the output features.

[0072] The target model parameters are obtained using the method described in the first aspect.

[0073] Fifthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein:

[0074] The processor and memory communicate with each other via a bus;

[0075] The memory stores program instructions that can be executed by the processor, and the processor can execute the first or second method by calling the program instructions.

[0076] Sixthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising:

[0077] A non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the methods in various possible implementations of the first or second aspect.

[0078] In a seventh aspect, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the methods in various possible implementations of the first or second aspect.

[0079] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0080] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 A schematic flowchart of a scheduling method for large model parameters provided in an embodiment of this application;

[0082] Figure 2 A large model architecture diagram provided for embodiments of this application;

[0083] Figure 3 A large model inference architecture diagram provided for embodiments of this application;

[0084] Figure 4 This is a schematic diagram of a large model inference method provided in an embodiment of this application;

[0085] Figure 5 A schematic diagram of a scheduling device for large model parameters provided in an embodiment of this application;

[0086] Figure 6 This is a schematic diagram of a large-scale model inference device provided in an embodiment of this application;

[0087] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0088] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0090] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0091] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0092] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0093] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0094] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0095] With the rapid iterative updates of large language models, sparse hybrid expert models (MoE) have attracted much attention as a novel model architecture. Introducing sparse hybrid expert models into large models aims to improve the model's expressive power and performance while maintaining computational efficiency.

[0096] Sparse expert models refer to models that do not use all the "expert" networks, but dynamically select a subset of experts to participate in the computation based on the input. This can significantly reduce computational costs while improving model performance. Assumption: Each transformer module has 256 fine-grained experts in its FFN module, and 8 experts are activated each time. This approach avoids the full computation of all experts in each inference, thus drastically reducing the computational load.

[0097] The large model with 671B parameters presents several challenges during inference. Firstly, memory requirements: a large model with 671B parameters requires enormous memory to store intermediate data such as parameters and activation values. Even with efficient memory management and distributed computing solutions, single-machine inference struggles to fully handle such a massive model. Secondly, latency issues: as the number of model parameters increases dramatically, the computational load for each inference iteration multiplies, leading to a significant increase in inference latency.

[0098] Research has revealed that in existing hybrid expert models, the activation patterns of experts in the FFN module of each transformer layer exhibit certain characteristics. Therefore, a small model can be trained to predict expert activation. This allows all expert weights to be stored in CPU memory, while activated experts are stored in GPU memory. When the inference service needs to activate an expert, it is then loaded from CPU memory to GPU memory. For extremely sparse hybrid expert models, such as large models with an expert activation ratio of 8 / 256, this prediction and scheduling method can significantly reduce the deployment cost of extremely fine-grained expert models.

[0099] This application provides a method for scheduling parameters of a large model. By setting a corresponding parameter prediction model in each transformer module of the large model, when that transformer module receives input features, the parameter prediction model can predict the expert model required for inference in the next transformer module. Then, if it is determined that the model parameters of the expert model are not in the GPU, the model parameters can be scheduled from the CPU to the GPU, providing a data foundation for inference in the next transformer module. Compared to storing all the model parameters of the expert model in the GPU, the solution provided in this application reduces the unnecessary use of GPU memory.

[0100] It should be noted that the large model parameter scheduling method provided in this application embodiment can be applied to electronic devices, including terminals and servers; the terminal can specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server can specifically be an application server or a web server. The electronic device runs the large model, parameter prediction model, cache prediction model, parameter scheduling model, cache scheduling model, etc. mentioned in the following embodiments.

[0101] Figure 1 This is a schematic flowchart of a scheduling method for large model parameters provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0102] Step 101: Obtain the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module.

[0103] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. Their core idea is to learn patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent. Commonly used large models include the GPT series, the BERT series, and generalized large models. This application does not limit the specific type of large model.

[0104] The large model includes multiple cascaded transformer modules (L layers in this embodiment). Each transformer module contains a parameter prediction model. It can be understood that each transformer module contains a parameter prediction model, or some transformer modules may contain parameter prediction models. Figure 2 A large model architecture diagram is provided for embodiments of this application, such as Figure 2 As shown. Each transformer module includes multiple expert models. It should be noted that a transformer module can also include a shared expert model. In practical applications, some transformer modules may also include multiple expert models. The model parameters of the shared expert model are always stored in GPU memory; therefore, the model parameters of the shared expert model do not require scheduling.

[0105] The parametric prediction model is used to predict the target expert model that needs to be activated for inference in the next layer of the transformer module. The input to the parametric prediction model is the input features of the current layer transformer module, which in turn are the output features of the previous layer transformer module. It can be understood that if the current layer transformer module is the first layer, then its input features are the output features of the modules in the larger model preceding this layer transformer module. The model architecture of the parametric prediction model can be constructed using MLP or transformer architectures, etc.

[0106] Step 102: Analyze the input features using the parameter prediction model corresponding to the current transformer module to obtain the target expert model required for inference in the next transformer module.

[0107] The parametric prediction model analyzes and predicts based on the input features of the current transformer module to obtain the target expert model required for inference in the next transformer module. Specifically, the parametric prediction model can output the activation probabilities of all expert models included in the next transformer module, and then determine the target expert model from among multiple expert models based on these probabilities. It should be noted that the parametric prediction model can be pre-trained before prediction.

[0108] Step 103: Determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model.

[0109] In the specific implementation process, the determination method is to check whether the model parameters corresponding to the target expert model are in the GPU's memory. If they are, scheduling is not required; otherwise, scheduling is required. The situation where the model parameters corresponding to the target expert model are not in GPU memory can include: all model parameters corresponding to the target expert model are not in GPU memory, or some model parameters corresponding to the target expert model are not in GPU memory.

[0110] Step 104: If necessary, generate a parameter scheduling strategy.

[0111] Parameter scheduling strategy instructs an electronic device on how to schedule target model parameters of a target expert model from CPU memory to GPU memory. This includes, for example, specifying the size of the data block to be scheduled, whether to delete model parameters already stored in the GPU, and how many model parameters to delete. Environmental information such as the current bandwidth on the electronic device can be considered when generating the scheduling strategy.

[0112] Step 105: According to the parameter scheduling strategy, schedule the target model parameters of the target expert model from the CPU memory to the GPU memory.

[0113] In the specific implementation process, after determining the parameter scheduling strategy, the target model parameters are obtained from CPU memory according to the parameter scheduling strategy and stored in GPU memory. It should be noted that if the model parameters corresponding to all target expert models are not in GPU memory, during scheduling, all model parameters corresponding to all target expert models are scheduled from CPU memory to GPU memory. If the model parameters corresponding to some target expert models are not in GPU memory, only these model parameters can be scheduled from CPU memory to GPU memory. Furthermore, after scheduling the model parameters, CPU memory can continue to store these model parameters; that is, CPU memory always stores the model parameters of the full set of expert models. Alternatively, these model parameters can be left unsold and scheduled to CPU memory when they are no longer needed in GPU memory.

[0114] This application embodiment uses the parameter prediction model corresponding to each transformer module to predict the input features, obtains the target expert model required for the inference of the next transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the GPU memory usage.

[0115] Based on the above embodiments, the input features are analyzed using the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference in the next layer transformer module, including:

[0116] The input features are fed into the parameter prediction model of the current layer transformer module to obtain the probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to characterize the probability that the expert model is the expert model required for inference in the next layer transformer module;

[0117] Based on probability values, a target expert model is selected from multiple expert models according to the expert model selection strategy.

[0118] The expert model selection strategies include:

[0119] The number of target expert models selected from multiple expert models is greater than a preset number; and / or,

[0120] Starting with the highest probability value, obtain expert models whose sum of probability values ​​is greater than a preset threshold.

[0121] In the specific implementation process, the probability of each expert model being activated in the output of the parameter prediction model can be expressed as: PE i (i = 0, 1, 2, M-1), and Where M represents the number of expert models. When selecting a target expert model from multiple expert models, the activation probabilities of each expert model can be sorted, and a preset threshold θ can be set. The preset number of expert models in the expert model selection strategy can be determined based on the number of expert models activated in each transformer module each time. For example, if the number of expert models activated each time is 8, then the preset number should be at least 8. To improve the model recall, the preset number can be set to 10, 11, etc. Furthermore, the sum of the probability values ​​of the selected expert models must be greater than the preset threshold θ. During selection, expert models can be sorted from largest to smallest probability value, starting with the highest probability value, so that the sum of the probability values ​​of the selected expert models is exactly greater than the preset threshold. That is, if one less expert model is selected, the sum of the probability values ​​will not meet the condition of being greater than the preset threshold. In other words, assuming n expert models are selected, the selected experts must satisfy the following constraints: and

[0122] This application embodiment selects a target expert model from multiple expert models through an expert model selection strategy. The number of selected target expert models changes dynamically according to the probability value of each expert model, which can improve the accuracy of expert model parameter scheduling.

[0123] Based on the above embodiments, a parameter scheduling strategy is generated, including:

[0124] The target model parameters, the model parameters already stored in the GPU memory, and the environmental features are input into the parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model. Among them, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM utilization rate, dynamic random access memory (DRAM) capacity, DRAM utilization rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization rate, and GPU utilization rate.

[0125] In the specific implementation process, the embodiments of this application introduce a parameter scheduling model, which takes the target model parameters and environmental characteristics as inputs and outputs a better scheduling strategy to guide the reading, writing, migration and elimination of target model parameters.

[0126] The target model parameters are obtained through the parameter prediction model described in the above embodiments. Environmental characteristics refer to the hardware and software environmental characteristics of the electronic device, including at least one of the following:

[0127] High-bandwidth memory (HBM) capacity: The total capacity of HBM reflects how much data the system can store.

[0128] HBM occupancy rate: The current usage of HBM, used to assess the remaining space of HBM.

[0129] Dynamic Random Access Memory (DRAM) Capacity: The total capacity of DRAM, which also reflects storage capacity.

[0130] DRAM utilization: Current DRAM usage, used to assess the remaining DRAM space.

[0131] CPU direct read / write bandwidth: The rate at which the CPU accesses memory (including HBM and DRAM), affecting data processing speed.

[0132] GPU direct read / write bandwidth: the speed at which the GPU accesses memory, which is particularly important for tasks such as graphics processing and deep learning.

[0133] CPU utilization: The current workload of the CPU, reflecting the usage of its processing power.

[0134] GPU utilization: The current workload of the GPU, which also reflects the usage of its processing power.

[0135] The target model parameters refer to the model parameters of the expert model required for inference in the next layer transformer module of the large model. The role of the model parameters already stored in the GPU memory is to determine whether the target model parameters have been stored in the GPU memory, thereby determining whether the target model parameters need to be scheduled.

[0136] The target model parameters, the model parameters already stored in the GPU, and environmental features are input into the scheduling model. The scheduling model can be a machine learning, deep learning, or reinforcement learning-based model that learns from the input data and outputs the optimal parameter scheduling strategy. In other words, it first determines whether the target model parameters need to be scheduled, and if so, how to schedule them.

[0137] The parameter scheduling model can contain multiple processing layers, such as input layer, hidden layer and output layer, which process input data and generate output through complex network structure and algorithm.

[0138] The parameter scheduling model outputs a scheduling strategy that may include the read / write order of the target model parameters, migration strategies, eviction rules, etc. The parameter scheduling strategy aims to optimize system performance, such as reducing access latency, increasing data throughput, or balancing resource usage. In this embodiment, the parameter scheduling model can consider current environmental characteristics, thereby allocating storage and computing resources more effectively and reducing resource waste; and it can be applied to heterogeneous systems with different types of memories and processors to fully utilize the overall system performance.

[0139] Based on the above embodiments, an adaptive reinforcement learning framework can be adopted for the parameter scheduling model. Adaptability refers to the ability to adapt to different hardware resources and different model sizes. In this embodiment, the parameter scheduling model can be adapted to different hardware resources after one reinforcement learning iteration. When the scheduling model is deployed on different hardware, it can be fine-tuned to optimize model performance.

[0140] The core idea of ​​reinforcement learning is that an agent learns optimal policies through interaction with its environment. Given a state, the agent chooses an action and observes the rewards or penalties given by the environment to adjust its policy in order to obtain greater cumulative rewards in the future. This process can be summarized as follows: the agent chooses an action based on the current state, the environment returns the next state and reward based on the action, and the agent updates its policy based on the reward.

[0141] The reinforcement learning in this application is not limited to classic value-based or policy-based reinforcement learning methods, such as Deep Q-Network (DQN), Asynchronous Advantage Actor Critic (A3C), Proximal Policy Optimization (PPO), etc.

[0142] Reinforcement learning focuses on the representation of the state space, reward function, and actions. The state space includes: the category of the large model framework used, the inferred tokens, the current token, the location information and storage location of the current token (including the storage locations of the inferred tokens and the current token), HBM capacity, HBM utilization, DRAM capacity, DRAM utilization, CPU and GPU direct read / write bandwidth and utilization, and the output models_needs of the pre-trained parameter prediction model (i.e., the target model parameters), etc.; it is not limited to heterogeneous storage media such as HBM and DRAM, nor is it limited to heterogeneous computing accelerators such as CPUs and GPUs. The characteristics of the state components are described as follows:

[0143] S=[Token_position, Token, whole_pre_tokens, whole_DRAM, DRAM_Usage, whole_HBM, HBM_Usage, Read_bandwidth, Write_bandwidth, Read_Usage_bandwidth, Write_Usage_bandwidth; models_needs].

[0144] Wherein, Token_position is the position information of the current token; Token is the current token; whole_pre_tokens is the inferred tokens; whole_DRAM is the DRAM capacity; DRAM_Usage is the DRAM utilization rate; whole_HBM is the HBM capacity; HBM_Usage is the HBM utilization rate; Read_bandwidth is the read bandwidth; Write_bandwidth is the write bandwidth; Read_Usage_bandwidth is the read utilization rate; Write_Usage_bandwidth is the write utilization rate; and tokens_needs is the target KV cache.

[0145] The action is the communication or holding action of the data required for the next-layer transformer module to infer the token. The reward function includes: the negative of the current inference loss, HBM occupancy, and bandwidth occupancy; this reward function can be expressed as:

[0146] Reward(s,index)=-loss+HBM_Usag+Read_Usag_bandwidth+Write_Usag_bandwidth

[0147] Where Reward(s,index) is the reward value; loss is the loss value of the current inference; HBM_Usag is the HBM utilization rate; Read_Usag_bandwidth is the read bandwidth utilization rate; and Write_Usag_bandwidth is the write bandwidth utilization rate.

[0148] The action space of reinforcement learning defines the operations that can be performed, mainly including: transfer operations and actions that maintain the original state; transfer operations include: moving a data block from HBM to DRAM; moving a data block from DRAM back to HBM.

[0149] In this embodiment, reinforcement learning algorithms enable parameter scheduling models to self-adjust and optimize based on environmental feedback. Through continuous learning, they can consider the occupancy and performance characteristics of various resources, thereby making more reasonable resource allocation strategies.

[0150] In the specific implementation process, before determining whether the target model parameters of the target expert model need to be scheduled based on the target expert model, the method also includes:

[0151] By analyzing the input features using the parameter prediction model corresponding to the current layer transformer module, the target expert model required for the current layer transformer module to infer the next token is obtained.

[0152] In practical implementation, the parameter prediction model can predict not only the target expert model required for inference in the next layer of the transformer module, but also the target expert model that the current layer of the transformer module needs to activate when inferring the next token. Therefore, the prediction of the target expert model that a certain layer of the transformer module needs to activate when inferring a certain token can be made by both the parameter prediction model of the current layer and the parameter prediction model of the previous layer. Thus, the target expert model can be the union of the target expert models predicted by the parameter prediction models of the current layer and the previous layer.

[0153] The parameter prediction model in this embodiment can not only predict the target expert model required for the next layer transformer module to infer, but also predict the target expert model required for the current layer transformer module to infer the next token. Therefore, the target expert model required for the transformer module to infer a token is predicted based on the parameter prediction model of the current layer and the parameter prediction model of the previous layer transformer module, thereby improving the accuracy of the prediction.

[0154] Based on the above embodiments, GPU memory has a limited capacity and cannot store data indefinitely. Therefore, when it is necessary to schedule target model parameters to GPU memory, deleting some model parameters from GPU memory is an effective solution if it is determined that GPU memory is insufficient. It should be noted that the criteria for determining whether GPU memory is insufficient can be preset. For example, when the GPU memory utilization rate reaches 80%, it can be considered that the storage space is insufficient; or, the remaining capacity of GPU memory is less than the size of the target model parameters to be scheduled.

[0155] When deleting model parameters, you can remove the least frequently accessed parameters. This can be achieved by maintaining an access counter or using an LRU (Least Recently Used) cache eviction policy. Alternatively, you can delete parameters based on importance or priority. For example, model parameters in GPU memory have their own priorities, and you can delete lower-priority parameters. You can also delete parameters based on their write time to GPU memory, starting with the earliest written parameter and continuing until subsequent target model parameters can be written to GPU memory. It should be noted that model parameters sharing an expert model will not be deleted.

[0156] There are several strategies for determining the number of model parameters to delete. For example, you can delete parameters in a preset size, such as deleting 5K parameters at a time. If this still doesn't satisfy the requirement of storing the target model parameters in GPU memory, then delete another 5K parameters until the requirement is met. Alternatively, you can delete model parameters from GPU memory in a manner that corresponds to the size of the target model parameters to be scheduled.

[0157] This application embodiment maintains the model parameters in GPU memory through deletion rules, so that the model parameters of the thermal expert model are always stored in GPU memory, reducing the number of model parameter scheduling times.

[0158] Based on the above embodiments, the method further includes:

[0159] The cache prediction model is used to predict the target key-value cache data required for the large model to infer subsequent tokens; subsequent tokens refer to tokens that the large model has not yet inferred.

[0160] Determine whether the target key-value cache data needs to be scheduled;

[0161] If scheduling is required, a cache scheduling strategy will be generated;

[0162] According to the cache scheduling policy, the target key-value cache data is obtained from the CPU memory and stored in the GPU memory. The target key-value cache data is used to enable the large model to infer subsequent tokens. The CPU memory is used to store the full amount of key-value cache data. During the inference process, the large model reads the required key-value cache data from the GPU memory.

[0163] In practical implementation, the large computational power and bandwidth required for the decoding stage of large model inference lead to wasted computing resources. To improve computational utilization, effectively reducing the memory-to-computation ratio is a crucial method. Researchers have successively reduced bandwidth requirements during the decoding stage through methods such as KV cache quantization and token sparsification. While these methods improve TBT (Total Target Bit Transmission), they also result in a loss of accuracy for large model inference. Subsequent research revealed that large models often do not require all token information during the decoding process of the inference service. Therefore, similar to the model parameter scheduling in the aforementioned embodiments, target key-value cache data that may be used later is cached and then scheduled into GPU memory for subsequent inference.

[0164] The cache prediction model is pre-trained, and its main function is to predict the target KV cache required for the tokens that the large model will infer. The prediction model can include pooling layers and fully connected layers. During training, data from common KV cache sparsity schemes can be used for pre-training of the small model; for example, generating token sequence data required for the large model's inference service process, such as Minference. Understandably, because the prediction model has a small parameter scale and requires less computing resources for inference, the parameter prediction model and the cache prediction model can be deployed on the CPU or GPU of an electronic device, or one can be deployed on the CPU and the other on the GPU. Since the amount of data output from the parameter prediction model and the cache prediction model to the large model is small, the impact on bandwidth is minimal. When considering scenarios with high GPU utilization, the parameter prediction model and the cache prediction model can be deployed on the CPU. The deployment location of the parameter prediction model and the cache prediction model can also be dynamically selected based on the actual load. For example, when the GPU load is low, the parameter prediction model and the cache prediction model can be migrated to the GPU to run, and when the GPU load is high, they can be switched to the CPU.

[0165] The subsequent token refers to a token that the large model has not yet reasoned about at the current moment. For example, if the large model is currently reasoning about the current token, then the subsequent token could be the next token that the large model is about to reason about, or it could be the two tokens following the current token, etc. For ease of description, the embodiments of this application refer to the subsequent token as the next token after the currently reasoned token.

[0166] After obtaining the target KV cache, the system determines whether scheduling of the target KV cache is necessary. The main method for this determination is to check if the target KV cache is present in GPU memory. If it is, scheduling is not required; otherwise, scheduling is required. However, even if GPU memory only contains a portion of the target KV cache, it is still considered as not containing it and scheduling is necessary. In subsequent scheduling, only the portion of the target KV cache not present in GPU memory is stored in GPU memory. For example, if the target KV cache includes KV caches corresponding to tokens 1-10, 30-50, and 100-110, and GPU memory currently stores KV caches for tokens 1-10 and 30-50, but not for tokens 100-110, then scheduling is required. During scheduling, the KV cache corresponding to tokens 100-110 is stored in GPU memory.

[0167] It is understood that GPU memory is the storage space of the target KV cache that the large model needs to read during inference. Specifically, it can be HBM in the GPU, or other storage space in the GPU; this application does not specifically limit this. Furthermore, the cache prediction model can output the index value of the target KV cache and use this index value to search in GPU memory for a KV cache with the same index value, thereby determining whether scheduling is needed. Moreover, when scheduling is needed, the corresponding KV cache can also be retrieved from CPU memory based on the index value and then stored in GPU memory.

[0168] Cache scheduling policies instruct electronic devices on how to schedule target key-value caches from CPU memory to GPU memory. For example, they specify the size of the data blocks to be scheduled, whether to delete key-value caches from GPU memory, and how many key-value caches to delete. When generating scheduling policies, environmental information such as the current bandwidth on the electronic device can be considered.

[0169] For cases where it is necessary to delete the KV cache within the GPU, the deletion strategy can refer to the deletion strategy for model parameters in the GPU in the above embodiments, which will not be repeated here.

[0170] After determining the cache scheduling strategy, the target KV cache is obtained from the CPU memory according to the cache scheduling strategy and stored in the GPU memory.

[0171] This application embodiment predicts the target key-value cache data required for subsequent token inference by using a prediction model, and performs operations such as scheduling judgment, scheduling strategy generation and data migration on the target KV cache, reducing frequent access to the first storage space. Furthermore, only the required target key-value cache data is transmitted each time, rather than the full amount of key-value cache data, which greatly reduces the bandwidth resource occupation caused by transmitting key-value cache data.

[0172] Based on the above embodiments, the target key-value cache data required for large model inference of subsequent tokens is predicted using a cache prediction model, including:

[0173] Input the already inferred tokens, the current token, and the location information of the current token into the cache prediction model to obtain the target key-value cache data required for subsequent token inference from the cache prediction model; where the current token refers to the token that the large model is inferring.

[0174] In practice, during large model inference, KVcache is usually used to store the hidden states or intermediate results that have already been calculated in order to improve inference efficiency and avoid repeated calculations.

[0175] Inferred tokens refer to tokens that have been inferred by the large model. Specifically, they can be represented in the form of a token sequence. The hidden state or intermediate results of these tokens have been stored in the KV cache.

[0176] The current token refers to the token that the large model is inferring. The position information of the current token refers to the position of the token that the large model is inferring in the inference service sequence. It can be represented by an index value, which is used to predict the order relationship of the tokens.

[0177] The already inferred tokens, the current token, and the position information of the current token are combined into a complete input sequence and input into the prediction model. The prediction model generates the target KV cache required by the large model for inference of the next token based on the input content.

[0178] This application embodiment utilizes a predictive model to analyze the reasoned tokens, the current token, and the location information of the current token, thereby capturing contextual information and more accurately predicting the target key-value cache data required for subsequent tokens.

[0179] Based on the above embodiments, a cache scheduling strategy is generated, including:

[0180] The target key-value cache data, the key-value cache data already stored in GPU memory, and environmental features are input into the cache scheduling model to obtain the cache scheduling policy output by the cache scheduling model. Among them, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM utilization rate, dynamic random access memory (DRAM) capacity, DRAM utilization rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization rate, and GPU utilization rate.

[0181] In the specific implementation process, the embodiments of this application introduce a cache scheduling model, take the target KV cache and environmental characteristics as input, and output a better cache scheduling strategy to guide the read, write, migration and eviction operations of the KV cache.

[0182] The target KV cache is obtained through prediction using the prediction model described in the above embodiments. Environmental characteristics refer to the hardware and software environmental characteristics of the electronic device, including at least one of the following: High Bandwidth Memory (HBM) capacity, HBM utilization, Dynamic Random Access Memory (DRAM) capacity, DRAM utilization, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization, and GPU utilization. Explanations of each of the above environmental characteristics are provided in the above embodiments and will not be repeated here. The method for generating a cache scheduling strategy based on the above environmental characteristics, the target key-value cache data, and the key-value cache data already stored in the GPU memory is similar to the method for generating a parameter scheduling strategy and will not be repeated here.

[0183] In this embodiment, the scheduling model can take into account the current environmental characteristics, thereby allocating storage and computing resources more effectively and reducing resource waste; and it can be applied to heterogeneous systems with different types of memory and processors to make full use of the overall system performance.

[0184] Figure 3 The large model inference architecture diagram provided for the embodiments of this application is as follows: Figure 3 As shown, this large model is deployed on electronic devices, including CPUs and GPUs. The CPU includes CPU computing cores and DRAM. The DRAM stores the full KV cache, and can also store parameter prediction models, parameter scheduling models, cache prediction models, and cache scheduling models. The GPU includes GPU computing cores and HBM. The HBM stores the weights of the large model's LLM, hot model parameters, and hot KV cache. It can be understood that hot model parameters refer to the expert model parameters frequently used during the large model's inference process. Hot KV cache refers to the KV cache frequently used during the large model's inference process, as well as the KV cache required for inferring subsequent tokens.

[0185] The scheduling of model parameters and KV cache can be implemented using the above-described embodiments, and will not be repeated here.

[0186] Figure 4 This is a schematic diagram of a large model inference method provided in an embodiment of this application, such as... Figure 4 As shown, the method includes:

[0187] Step 401: Input the reasoned tokens into the large model; the large model consists of multiple transformer modules, and each transformer module contains multiple expert models;

[0188] Step 402: Determine the active expert model from multiple expert models using the input features of each transformer module;

[0189] Step 403: For cases where the target model parameters stored in GPU memory contain the activation model parameters corresponding to all activation expert models, infer the input features based on the activation model parameters corresponding to the activation expert models to obtain the output features;

[0190] The target model parameters are obtained using the method described in the above embodiments.

[0191] In the specific implementation process, the inferred tokens are input into the large model, and inference is performed layer by layer through the transformer modules of the large model. During the inference process of each transformer module, since each transformer module includes multiple expert models, the router selects the expert model to be activated based on the input features; in this embodiment, this is referred to as activating the expert model. When analyzing and inferring the input features through the activated expert model, the model parameters of the activated expert model need to be obtained from the GPU memory. If the GPU memory contains the model parameters of all activated expert models, inference can be performed after obtaining the model parameters. The result obtained after inference by the current transformer module is input into the next transformer module, and then inference is performed again according to the above method, finally obtaining the output features of the large model.

[0192] This application embodiment uses the parameter prediction model corresponding to each transformer module to predict the input features, obtains the target expert model required for the inference of the next transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the GPU memory usage.

[0193] Building upon the above embodiments, since the model parameters stored in GPU memory are obtained based on predictions made by parametric prediction models, and the accuracy of these predictions is not 100%, the model parameters of the expert models that need to be activated during inference may not be in GPU memory, while CPU memory always stores the model parameters of all expert models. To address this, inference can be performed on the input features based on a subset of the activated model parameters stored in GPU memory and the remaining activated model parameters stored in CPU memory to obtain the output features.

[0194] In the specific implementation process, for the activation model parameters of the partial activation expert models already stored in GPU memory, these activation expert models can continue inference on the GPU. For activation model parameters not stored in GPU memory, the input features can be input into CPU memory, and inference can be performed using the activation expert models corresponding to the remaining activation model parameters in the CPU. Finally, the output results of the activation expert models in the GPU and the CPU are integrated to obtain the final result.

[0195] In this embodiment, the target expert model predicted by the parameter prediction model may be inaccurate. That is, the GPU may not contain all the model parameters of the activated expert model required for inference of a certain transformer layer containing a large model. In this case, the GPU and CPU can be connected to perform inference together, without having to schedule the remaining activated expert model parameters from the CPU to the GPU again, thus improving the model inference efficiency.

[0196] Based on the above embodiments, the method further includes:

[0197] The target key-value cache data of the token to be inferred is obtained through a large model; wherein, the target key-value cache data is obtained by using the scheduling method of the large model parameters described in the above embodiment;

[0198] Correspondingly, the partial activation model parameters based on GPU memory storage include:

[0199] Inference is performed on the input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

[0200] In the specific implementation process, in addition to activating the model parameters of the expert model, the large model also needs to rely on the target key-value cache data during the inference process. Therefore, the large model can perform inference on the input features based on the model parameters corresponding to the activated expert model and the target key-value cache data stored in the GPU memory.

[0201] This application embodiment performs inference on input features based on the model parameters corresponding to the activated expert model and the target key-value cache data. The model parameters corresponding to the activated expert model and the target key-value cache data are obtained from targeted scheduling of the CPU, which improves inference efficiency while reducing frequent access to the first storage space.

[0202] Furthermore, large model inference, parameter and cache prediction, and scheduling can be executed asynchronously. For example, during the large model's inference of the current token, the cache prediction model can predict the target KV cache required for the large model's inference of the next token. If the target KV cache is already stored in GPU memory, no scheduling is needed; if the target KV cache is not in GPU memory, the cache scheduling model will schedule the target KV cache from CPU memory to GPU memory. When the large model infers the next token, it can directly read the target KV cache from GPU memory. At this time, the prediction model can predict the target KV cache required by the large model for inference of the token after that, and so on. Moreover, during the inference process of a certain transformer module in the large model, the parameter prediction module of that layer can predict the target model parameters required for the inference of the next transformer module, as well as the target model parameters required by that transformer module for inferring the next token. If the target model parameters are not in GPU memory, they will be scheduled from CPU memory to GPU memory. By executing asynchronously, when a large model is inferring a certain token, it can directly read the target KV cache and activation model parameters required for the token inference from the GPU, saving the time of reading the full KV cache and activation model parameters from the CPU and improving inference efficiency; it also reduces the number of KV cache and activation model parameters read, thus reducing bandwidth consumption.

[0203] Figure 5 This is a schematic diagram of a scheduling device for large model parameters provided in an embodiment of this application. This device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the various steps of the method embodiment can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The device includes: a feature acquisition module 501, an expert model prediction module 502, a judgment module 503, a scheduling strategy generation module 504, and a scheduling module 505, wherein:

[0204] The feature acquisition module 501 is used to acquire the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module.

[0205] The expert model prediction module 502 is used to analyze the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the inference of the next layer transformer module.

[0206] The judgment module 503 is used to determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model;

[0207] The scheduling strategy generation module 504 is used to generate a parameter scheduling strategy when it is necessary to schedule the target model parameters;

[0208] The scheduling module 505 is used to schedule the target model parameters of the target expert model from CPU memory to GPU memory according to the parameter scheduling strategy.

[0209] Based on the above embodiments, the determination module 503 is specifically used for:

[0210] Determine whether the target model parameters of the target expert model are stored in the GPU memory;

[0211] If the target model parameters are not stored in the GPU memory, then scheduling is required; otherwise, scheduling is not required.

[0212] Based on the above embodiments, the expert model prediction module 502 is specifically used for:

[0213] The input features are input into the parameter prediction model of the current layer transformer module to obtain the probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to characterize the probability that the expert model is the expert model required for inference of the next layer transformer module;

[0214] Based on the probability value, the target expert model is selected from multiple expert models according to the expert model selection strategy;

[0215] The expert model selection strategy includes:

[0216] The number of target expert models selected from the plurality of expert models is greater than a preset number; and / or,

[0217] Starting with the highest probability value, obtain expert models whose sum of probability values ​​is greater than a preset threshold.

[0218] Based on the above embodiments, the scheduling strategy generation module 504 is specifically used for:

[0219] The target model parameters, the model parameters already stored in the GPU memory, and the environmental features are input into the parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model; wherein, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM occupancy rate, dynamic random access memory (DRAM) capacity, DRAM occupancy rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU occupancy rate, and GPU occupancy rate.

[0220] Based on the above embodiments, the parameter scheduling model is obtained by training based on a reinforcement learning algorithm.

[0221] Based on the above embodiments, the expert model prediction module 502 is also used for:

[0222] The input features are analyzed by the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the current layer transformer module to infer the next token.

[0223] Based on the above embodiments, the device further includes a deletion module, used for:

[0224] If the GPU memory is insufficient, some model parameters are deleted from the GPU memory based on the deletion rules.

[0225] Based on the above embodiments, the device further includes a cache prefetch module, used for:

[0226] The cache prediction model is used to predict the target key-value cache data required for the large model to infer subsequent tokens; the subsequent tokens refer to the tokens that the large model has not yet inferred.

[0227] Determine whether the target key-value cache data needs to be scheduled;

[0228] If scheduling is required, a cache scheduling strategy will be generated;

[0229] According to the cache scheduling policy, the target key-value cache data is obtained from the CPU memory and stored in the GPU memory; wherein, the target key-value cache data is used to enable the large model to infer the subsequent token; the CPU memory is used to store the full amount of key-value cache data; during the inference process, the large model reads the required key-value cache data from the GPU memory.

[0230] Based on the above embodiments, the cache prefetch module is specifically used for:

[0231] The already inferred tokens, the current token, and the location information of the current token are input into the cache prediction model to obtain the target key-value cache data required for the subsequent token inference output by the cache prediction model; wherein, the current token refers to the token that the large model is inferring.

[0232] Based on the above embodiments, the cache prefetch module is specifically used for:

[0233] The target key-value cache data, the key-value cache data already stored in the GPU memory, and environmental features are input into the cache scheduling model to obtain the cache scheduling strategy output by the cache scheduling model; wherein, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM utilization rate, dynamic random access memory (DRAM) capacity, DRAM utilization rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization rate, and GPU utilization rate.

[0234] Figure 6 This is a schematic diagram of a large model inference device provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 4 The method implementation corresponds to this and can be executed. Figure 4 The various steps involved in the method embodiment, and the specific functions of the device, can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes: an input module 601, an expert model determination module 602, and an inference module 603, wherein:

[0235] Input module 601 is used to input the reasoned tokens into the large model; the large model includes multiple transformer modules, and each transformer module includes multiple expert models;

[0236] The expert model determination module 602 is used to determine the active expert model from multiple expert models using the input features of the transformer module at each layer;

[0237] The inference module 603 is used to infer the input features based on the activation model parameters corresponding to the activation expert models when the target model parameters stored in GPU memory contain all the activation model parameters corresponding to the activation expert models, and obtain the output features.

[0238] The target model parameters are obtained using the method described in the above embodiments.

[0239] Based on the above embodiments, the inference module 603 is further configured to:

[0240] In cases where the target model parameters stored in the GPU memory contain some of the activation model parameters corresponding to the activation expert model, inference is performed on the input features based on some of the activation model parameters stored in the GPU memory and other activation model parameters stored in the CPU memory.

[0241] The CPU memory stores the model parameters corresponding to all expert models.

[0242] Based on the above embodiments, the cached data acquisition module is used for:

[0243] The target key-value cache data of the token to be inferred is obtained through a large model; wherein, the target key-value cache data is obtained by using the scheduling method of the large model parameters described in the above embodiment;

[0244] Accordingly, the inference module 603 is specifically used for:

[0245] Inference is performed on the input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

[0246] Based on the above embodiments, the inference process of the large model, the scheduling process of the target model parameters, and the scheduling process of the target key-value cache data are executed asynchronously.

[0247] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device includes: a processor 701, a memory 702, and a bus 703; wherein:

[0248] The processor 701 and the memory 702 communicate with each other via the bus 703;

[0249] The processor 701 is used to call program instructions in the memory 702 to execute the methods provided in the above-described method embodiments, such as: obtaining the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module; analyzing the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference by the next layer transformer module; determining whether it is necessary to schedule the target model parameters of the target expert model based on the target expert model; if so, generating a parameter scheduling strategy; and scheduling the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy. Alternatively,

[0250] The inferred tokens are input into the large model; the large model includes multiple transformer modules, and each transformer module includes multiple expert models; the activated expert model is determined from the multiple expert models based on the input features of each transformer module; for the case where the target model parameters stored in GPU memory contain the activated model parameters corresponding to all activated expert models, the input features are inferred based on the activated model parameters corresponding to the activated expert models to obtain the output features; wherein, the target model parameters are obtained using the method of the above embodiment.

[0251] The processor 701 can be an integrated circuit chip with signal processing capabilities. The processor 701 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0252] The memory 702 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0253] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by the computer, the computer can execute the methods provided in the above-described method embodiments, such as: obtaining the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module; analyzing the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference by the next layer transformer module; determining whether it is necessary to schedule the target model parameters of the target expert model based on the target expert model; if so, generating a parameter scheduling strategy; and scheduling the target model parameters of the target expert model from the central processing unit (CPU) memory to the image processing unit (GPU) memory according to the parameter scheduling strategy. Alternatively,

[0254] The inferred tokens are input into the large model; the large model includes multiple transformer modules, and each transformer module includes multiple expert models; the activated expert model is determined from the multiple expert models based on the input features of each transformer module; for the case where the target model parameters stored in GPU memory contain the activated model parameters corresponding to all activated expert models, the input features are inferred based on the activated model parameters corresponding to the activated expert models to obtain the output features; wherein, the target model parameters are obtained using the method of the above embodiment.

[0255] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions. These instructions cause a computer to execute the methods provided in the above-described method embodiments, including, for example: obtaining the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module; analyzing the input features using the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference in the next layer transformer module; determining whether it is necessary to schedule the target model parameters of the target expert model based on the target expert model; if so, generating a parameter scheduling strategy; and scheduling the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy. Alternatively,

[0256] The inferred tokens are input into the large model; the large model includes multiple transformer modules, and each transformer module includes multiple expert models; the activated expert model is determined from the multiple expert models based on the input features of each transformer module; for the case where the target model parameters stored in GPU memory contain the activated model parameters corresponding to all activated expert models, the input features are inferred based on the activated model parameters corresponding to the activated expert models to obtain the output features; wherein, the target model parameters are obtained using the method of the above embodiment.

[0257] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0258] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0259] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0260] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0261] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A scheduling method for large model parameters, characterized in that, The large model includes multiple transformer modules, and a parameter prediction model is set for one of the transformer modules. The method includes: Obtain the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module; The input features are analyzed by the parameter prediction model corresponding to the current transformer module to obtain the target expert model required for inference in the next transformer module; Based on the target expert model, determine whether the target model parameters of the target expert model need to be scheduled; If necessary, generate a parameter scheduling strategy; According to the parameter scheduling strategy, the target model parameters of the target expert model are scheduled from the CPU memory to the GPU memory.

2. The method according to claim 1, characterized in that, The step of determining whether to schedule the target model parameters based on the target expert model includes: Determine whether the target model parameters of the target expert model are stored in the GPU memory; If the target model parameters are not stored in the GPU memory, then scheduling is required; otherwise, scheduling is not required.

3. The method according to claim 1, characterized in that, The step of analyzing the input features using the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference in the next layer transformer module includes: The input features are input into the parameter prediction model of the current layer transformer module to obtain the probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to characterize the probability that the expert model is the expert model required for inference of the next layer transformer module; Based on the probability value, the target expert model is selected from multiple expert models according to the expert model selection strategy.

4. The method according to claim 3, characterized in that, The expert model selection strategy includes: The number of target expert models selected from the plurality of expert models is greater than a preset number; and / or, Starting with the highest probability value, obtain expert models whose sum of probability values ​​is greater than a preset threshold.

5. The method according to claim 1, characterized in that, The parameter generation scheduling strategy includes: The target model parameters, the model parameters already stored in the GPU memory, and the environmental features are input into the parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model; wherein, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM occupancy rate, dynamic random access memory (DRAM) capacity, DRAM occupancy rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU occupancy rate, and GPU occupancy rate.

6. The method according to claim 5, characterized in that, The parameter scheduling model was obtained by training based on a reinforcement learning algorithm.

7. The method according to claim 1, characterized in that, Before determining whether the target model parameters of the target expert model need to be scheduled based on the target expert model, the method further includes: The input features are analyzed by the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the current layer transformer module to infer the next token.

8. The method according to claim 1, characterized in that, Before scheduling the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy, the method further includes: If the GPU memory is insufficient, some model parameters are deleted from the GPU memory based on the deletion rules.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The cache prediction model is used to predict the target key-value cache data required for the large model to infer subsequent tokens; the subsequent tokens refer to the tokens that the large model has not yet inferred. Determine whether the target key-value cache data needs to be scheduled; If scheduling is required, a cache scheduling strategy will be generated; According to the cache scheduling policy, the target key-value cache data is obtained from the CPU memory and stored in the GPU memory; wherein, the target key-value cache data is used to enable the large model to infer the subsequent token; the CPU memory is used to store the full amount of key-value cache data; during the inference process, the large model reads the required key-value cache data from the GPU memory.

10. The method according to claim 9, characterized in that, The method of using a cache prediction model to predict the target key-value cache data required for large model inference of subsequent tokens includes: The already inferred tokens, the current token, and the location information of the current token are input into the cache prediction model to obtain the target key-value cache data required for the subsequent token inference output by the cache prediction model; wherein, the current token refers to the token that the large model is inferring.

11. The method according to claim 9, characterized in that, The generation cache scheduling strategy includes: The target key-value cache data, the key-value cache data already stored in the GPU memory, and environmental features are input into the cache scheduling model to obtain the cache scheduling strategy output by the cache scheduling model; wherein, the environmental features include at least one of the following: high bandwidth memory (HBM) capacity, HBM utilization rate, dynamic random access memory (DRAM) capacity, DRAM utilization rate, CPU direct read / write bandwidth, GPU direct read / write bandwidth, CPU utilization rate, and GPU utilization rate.

12. A large-scale model reasoning method, characterized in that, The large model includes multiple transformer modules, each of which has a parameter prediction model, including: Input the reasoned tokens into the large model; the large model includes multiple transformer modules, and each transformer module includes multiple expert models. The active expert model is determined from multiple expert models by using the input features of the transformer module in each layer; For cases where the target model parameters stored in GPU memory contain all the activation model parameters corresponding to the activation expert models, the input features are inferred based on the activation model parameters corresponding to the activation expert models to obtain the output features. The target model parameters are obtained using the method described in any one of claims 1-8.

13. The method according to claim 12, characterized in that, The method further includes: In cases where the target model parameters stored in the GPU memory contain some of the activation model parameters corresponding to the activation expert model, inference is performed on the input features based on some of the activation model parameters stored in the GPU memory and other activation model parameters stored in the CPU memory. The CPU memory stores the model parameters corresponding to all expert models.

14. The method according to claim 12, characterized in that, The method further includes: The target key-value cache data of the token to be inferred is obtained through a large model; wherein the target key-value cache data is obtained by the scheduling method of the large model parameters as described in any one of claims 9-11; Accordingly, the inference based on the activation model parameters corresponding to the activation expert model on the input features includes: Inference is performed on the input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

15. The method according to claim 14, characterized in that, The inference process of the large model, the scheduling process of the target model parameters, and the scheduling process of the target key-value cache data are executed asynchronously.

16. A scheduling device for large model parameters, characterized in that, The large model includes multiple transformer modules and a parameter prediction model corresponding to each transformer module. The device includes: The feature acquisition module is used to acquire the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module. The expert model prediction module is used to analyze the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference of the next layer transformer module. The judgment module is used to determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model. The scheduling strategy generation module is used to generate a parameter scheduling strategy when it is necessary to schedule the target model parameters; The scheduling module is used to schedule the target model parameters of the target expert model from CPU memory to GPU memory according to the parameter scheduling strategy.

17. A large-scale model reasoning device, characterized in that, include: The input module is used to input the reasoned tokens into the large model; the large model includes multiple transformer modules, and each transformer module includes multiple expert models. An expert model determination module is used to determine the active expert model from multiple expert models using the input features of the transformer module at each layer; The inference module is used to infer the input features based on the activation model parameters corresponding to the activation expert models when the target model parameters stored in GPU memory contain all the activation model parameters corresponding to the activation expert models, and to obtain the output features. The target model parameters are obtained using the method described in any one of claims 1-8.

18. An electronic device, characterized in that, include: Processor, memory, and bus, among which: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can invoke the program instructions to perform the method as described in any one of claims 1-15.

19. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-15.

20. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-15.

Images