Large model parameter scheduling method and device, large model parameter reasoning method and device and electronic equipment

Abstract

The invention provides a large model parameter scheduling method, a large model parameter reasoning method, a large model parameter scheduling device, a large model parameter reasoning device and electronic equipment. The method comprises the following steps: acquiring an input feature of a current layer transformer module; the input feature is the feature output by the former layer of transformer module; the input features are analyzed through a parameter prediction model corresponding to the current layer transformer module, and a target expert model needed by reasoning of the next layer transformer module is obtained; if it is judged that target model parameters need to be scheduled based on the target expert model, a parameter scheduling strategy is generated; and scheduling the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy. According to the method and the device, the target expert model required by reasoning of the next layer of transformer module is obtained through the parameter prediction model, and the target model parameters are scheduled from the CPU to the GPU, so that the occupation of a GPU memory is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a scheduling method, reasoning method, device, and electronic device for large model parameters. Background Art

[0002] With the rapid iteration and update of large language models, sparse mixture of experts (MoE) models have attracted much attention as a new model architecture. The purpose of introducing sparse mixture of experts (MoE) models into large models is to improve the model's expressiveness and performance while maintaining computational efficiency.

[0003] During the large model inference process, the model parameters of the expert model are written into the GPU, and then the GPU is used for inference. Since the number of expert models activated in each layer of inference module is small, only a very small part of the model parameters written into the GPU are required for inference, resulting in a waste of GPU memory resources. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a scheduling method, inference method, device and electronic device for large model parameters to reduce the waste of GPU memory resources.

[0005] In a first aspect, an embodiment of the present application provides a method for scheduling parameters of a large model, wherein the large model includes multiple layers of transformer modules, and a parameter prediction model is provided for each transformer module. The method includes:

[0006] Get the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module;

[0007] Analyze the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for reasoning of the next layer transformer module;

[0008] Determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model;

[0009] If necessary, generate parameter scheduling strategy;

[0010] The target model parameters of the target expert model are scheduled from the central processing unit (CPU) memory to the image processing unit (GPU) memory according to the parameter scheduling strategy.

[0011] The embodiment of the present application predicts the input features by using the parameter prediction model corresponding to each layer of transformer module, obtains the target expert model required for reasoning of the next layer of transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the occupancy of the GPU memory.

[0012] In a possible implementation of the first aspect, determining whether target model parameters of the target expert model need to be scheduled based on the target expert model includes:

[0013] Determine whether the target model parameters of the target expert model are stored in the GPU memory;

[0014] If the target model parameters are not stored in the GPU memory, scheduling is required; otherwise, scheduling is not required.

[0015] The embodiment of the present application determines whether the target model parameters need to be scheduled by judging whether the target model parameters are stored in the GPU memory, thereby reducing unnecessary scheduling operations and reducing the overhead and time cost of data migration.

[0016] In one possible implementation of the first aspect, analyzing input features using a parameter prediction model corresponding to a transformer module in a current layer to obtain a target expert model required for reasoning in a transformer module in a next layer includes:

[0017] Input the input features into the parameter prediction model of the current layer transformer module to obtain the probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to represent the probability that the expert model is the expert model required for inference of the next layer transformer module;

[0018] Based on the probability value, the target expert model is selected from multiple expert models according to the expert model selection strategy.

[0019] In a possible implementation of the first aspect, the expert model selection strategy includes:

[0020] The number of target expert models selected from the plurality of expert models is greater than a preset number; and / or,

[0021] Starting from the maximum probability value, obtain the expert model whose sum of probability values is greater than the preset threshold.

[0022] The embodiment of the present application selects a target expert model from multiple expert models through an expert model selection strategy. The number of selected target expert models changes dynamically according to the probability value of each expert model, which can improve the accuracy of expert model parameter scheduling.

[0023] In a possible implementation of the first aspect, generating a parameter scheduling policy includes:

[0024] The target model parameters, the model parameters stored in the GPU memory, and the environmental characteristics are input into the parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model; wherein the environmental characteristics include at least one of the high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, central processing unit (CPU) direct read and write bandwidth, graphics processing unit (GPU) direct read and write bandwidth, CPU occupancy, and GPU occupancy.

[0025] The embodiments of the present application can allocate storage and computing resources more efficiently and reduce resource waste by taking into account the current environmental characteristics; and can be applied to heterogeneous systems with different types of memories and processors to fully utilize the overall performance of the system.

[0026] In a possible implementation of the first aspect, the parameter scheduling model is obtained through training based on a reinforcement learning algorithm.

[0027] In the embodiment of the present application, the reinforcement learning algorithm enables the model to self-adjust and optimize according to the feedback from the environment. Through continuous learning, it can consider the occupancy and performance characteristics of various resources, thereby making a more reasonable resource allocation strategy.

[0028] In a possible implementation of the first aspect, before determining whether target model parameters of the target expert model need to be scheduled based on the target expert model, the method further includes:

[0029] The input features are analyzed through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the current layer transformer module to infer the next token.

[0030] The parameter prediction model in the embodiment of the present application can not only predict the target expert model required for the next-layer transformer module reasoning, but also predict the target expert model required for the current-layer transformer module to reason about the next token. Therefore, the target expert model required for the transformer module to reason about a token is obtained by respectively predicting the parameter prediction model of the current layer and the parameter prediction model of the previous layer transformer module, thereby improving the accuracy of the prediction.

[0031] In a possible implementation of the first aspect, before scheduling the target model parameters of the target expert model from a central processing unit (CPU) memory to a graphics processing unit (GPU) memory according to the parameter scheduling strategy, the method further includes:

[0032] If the GPU memory is insufficient, some model parameters are deleted from the GPU memory based on the deletion rule.

[0033] The embodiment of the present application maintains the model parameters in the GPU memory by deleting rules, so that the model parameters of the thermal expert model are always stored in the GPU memory, reducing the number of model parameter scheduling times.

[0034] In a possible implementation of the first aspect, the method further includes:

[0035] Use the cached prediction model to predict the target key-value cache data required for the large model to infer subsequent tokens; subsequent tokens refer to tokens that have not yet been inferred by the large model;

[0036] Determine whether the target key-value cache data needs to be scheduled;

[0037] If scheduling is required, generate a cache scheduling strategy;

[0038] According to the cache scheduling strategy, the target key-value cache data is obtained from the CPU memory and stored in the GPU memory. The target key-value cache data is used to enable the large model to infer subsequent tokens. The CPU memory is used to store the full amount of key-value cache data. During the inference process, the large model reads the required key-value cache data from the GPU memory.

[0039] The embodiment of the present application predicts the target key-value cache data required for the subsequent token reasoning of the large model through a prediction model, and performs scheduling judgment, scheduling strategy generation, and data migration on the target key-value cache data, thereby reducing frequent access to the first storage space. In addition, only the required target key-value cache data is transmitted each time instead of the full amount of key-value cache data, which greatly reduces the bandwidth resource occupation caused by transmitting key-value cache data.

[0040] In one possible implementation of the first aspect, using a cache prediction model to predict target key-value cache data required for a large model to infer subsequent tokens includes:

[0041] Input the inferred tokens, current token, and the location information of the current token into the cache prediction model to obtain the target key-value cache data required for subsequent token inference output by the cache prediction model; among them, the current token refers to the token being inferred by the large model.

[0042] The embodiment of the present application uses a prediction model to analyze the inferred tokens, the current token, and the location information of the current token, so as to capture context information and more accurately predict the target key-value cache data required for subsequent tokens.

[0043] In a possible implementation of the first aspect, generating a cache scheduling policy includes:

[0044] The target key-value cache data, the key-value cache data stored in the GPU memory, and the environmental characteristics are input into the cache scheduling model to obtain a cache scheduling policy output by the cache scheduling model; wherein the environmental characteristics include at least one of high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, CPU direct read and write bandwidth, GPU direct read and write bandwidth, CPU occupancy, and GPU occupancy.

[0045] In the embodiment of the present application, the scheduling model can take into account the current environmental characteristics, so that storage and computing resources can be allocated more efficiently, reducing resource waste; and can be applied to heterogeneous systems with different types of memory and processors to fully utilize the overall performance of the system.

[0046] In a second aspect, an embodiment of the present application provides a large model inference method, wherein the large model includes multiple layers of transformer modules, each layer of transformer modules is provided with a parameter prediction model, including:

[0047] Input the inferred tokens into the large model; the large model includes multiple layers of transformer modules, each layer of transformer modules includes multiple expert models;

[0048] Determine the activated expert model from multiple expert models through the input features of each layer of transformer module;

[0049] In the case where the target model parameters stored in the GPU memory include the activation model parameters corresponding to all activated expert models, the input features are inferred based on the activation model parameters corresponding to the activated expert models to obtain the output features;

[0050] The target model parameters are obtained using the method of the first aspect.

[0051] The embodiment of the present application predicts the input features by using the parameter prediction model corresponding to each layer of transformer module, obtains the target expert model required for reasoning of the next layer of transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the occupancy of the GPU memory.

[0052] In a possible implementation of the first aspect, the method further includes:

[0053] When the target model parameters stored in GPU memory include activation model parameters corresponding to some activated expert models, the input features are inferred based on the partially activated model parameters stored in GPU memory and the remaining activation model parameters stored in CPU memory.

[0054] Among them, the model parameters corresponding to all expert models are stored in the CPU memory.

[0055] In an embodiment of the present application, the target expert model predicted by the parameter prediction model may be inaccurate, that is, the GPU may not contain all the model parameters of the activated expert model required for reasoning of a certain transformer layer of the large model. In this case, the GPU and CPU can be connected to jointly perform reasoning without the need to schedule the remaining model parameters of the activated expert model from the CPU to the GPU again, thereby improving the efficiency of model reasoning.

[0056] In a possible implementation of the first aspect, the method further includes:

[0057] Obtain target key-value cache data of the token to be inferred through the large model; wherein the target key-value cache data is obtained by using the scheduling method of the large model parameters of the first aspect;

[0058] Accordingly, the input features are inferred based on the activation model parameters corresponding to the activated expert model, including:

[0059] Reasoning about input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

[0060] The embodiment of the present application infers input features based on the model parameters and target key-value cache data corresponding to the activated expert model, wherein the model parameters and target key-value cache data corresponding to the activated expert model are obtained through targeted scheduling from the CPU, thereby reducing frequent access to the first storage space and improving reasoning efficiency.

[0061] In a possible implementation of the first aspect, the inference process of the large model and the scheduling process of the target model parameters and the scheduling process of the target key-value cache data are executed asynchronously.

[0062] The embodiment of the present application executes the reasoning of the large model and the scheduling of the target model parameters, as well as the scheduling of the target key-value cache data, in an asynchronous manner. During the reasoning process of the current layer transformer module, the target expert model required for the next layer transformer module reasoning is scheduled through the parameter prediction model of the current layer, and the corresponding model parameters are scheduled, and the target key-value cache data required for subsequent tokens are predicted and scheduled. There is no need to temporarily schedule the model parameters and target key-value cache data of the target expert model, thereby improving the reasoning efficiency of the large model.

[0063] In a third aspect, an embodiment of the present application provides a device for scheduling parameters of a large model, wherein the large model includes multiple layers of transformer modules and a parameter prediction model corresponding to each layer of transformer modules, and the device includes:

[0064] The feature acquisition module is used to obtain the input features of the transformer module in the current layer; the input features are the features output by the transformer module in the previous layer;

[0065] The expert model prediction module is used to analyze the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for inference of the next layer transformer module;

[0066] A judgment module, used for judging whether the target model parameters of the target expert model need to be scheduled based on the target expert model;

[0067] The scheduling strategy generation module is used to generate a parameter scheduling strategy when the target model parameters need to be scheduled;

[0068] The scheduling module is used to schedule the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy.

[0069] In a fourth aspect, an embodiment of the present application provides a large model reasoning device, wherein the large model includes multiple layers of transformer modules, and each layer of transformer modules includes multiple expert models;

[0070] An expert model determination module is used to determine the activated expert model from multiple expert models based on the input features of each layer of transformer module;

[0071] An inference module is used to infer input features based on the activation model parameters corresponding to the activated expert models when the target model parameters stored in the GPU memory contain the activation model parameters corresponding to all activated expert models, so as to obtain output features;

[0072] The target model parameters are obtained using the method of the first aspect.

[0073] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0074] The processor and memory communicate with each other through the bus;

[0075] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method of the first aspect or the second aspect.

[0076] In a sixth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including:

[0077] The non-transitory computer-readable storage medium stores computer instructions, which enable a computer to execute the method in each possible implementation of the first aspect or the second aspect.

[0078] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions. When the computer program instructions are read and executed by a processor, the methods in each possible implementation of the first aspect or the second aspect are executed.

[0079] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 A flow chart of a method for scheduling large model parameters provided in an embodiment of the present application;

[0082] Figure 2 A large model architecture diagram provided for an embodiment of the present application;

[0083] Figure 3 A diagram of the large model reasoning architecture provided in the embodiments of this application;

[0084] Figure 4 A flowchart of a large model reasoning method provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of the structure of a large model parameter scheduling device provided in an embodiment of the present application;

[0086] Figure 6 A schematic diagram of the structure of a large model inference device provided in an embodiment of the present application;

[0087] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0090] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0091] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0092] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0093] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0094] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0095] With the rapid iteration and update of large language models, sparse mixture of experts (MoE) models have attracted much attention as a new model architecture. When introducing sparse mixture of experts models into large models, the goal is to improve the model's expressiveness and performance while maintaining computational efficiency.

[0096] A sparse expert model doesn't utilize all "expert" networks. Instead, it dynamically selects a subset of experts based on the input. This significantly reduces computational costs while improving model performance. For example, each transformer layer's FFN module has 256 fine-grained experts, with only eight of them activated at a time. This approach avoids the full computation of all experts during each inference, significantly reducing computational effort.

[0097] The inference process for a large model with 671 Bytes of model parameters presents several challenges. First, memory requirements: A large model with 671 Bytes of model parameters requires a vast amount of memory to store intermediate data such as parameters and activation values. Even with efficient memory management and distributed computing solutions, single-machine inference cannot fully support such a large model. Second, there is the issue of latency: As the number of model parameters increases, the computational effort for each inference increases exponentially, significantly increasing inference latency.

[0098] Research has found that in existing hybrid expert models, the expert activations in the FFN modules of each transformer layer exhibit certain characteristics. Therefore, a small model can be trained to predict expert activations. This allows all expert weight parameters to be stored in CPU memory, while the activated experts are stored in video memory. When the inference service needs to activate the expert, it is loaded from CPU memory to GPU memory. This prediction and scheduling approach can significantly reduce the deployment cost of extremely fine-grained expert models for extremely sparse hybrid expert models, such as large models with an expert activation ratio of 8 / 256.

[0099] The embodiment of the present application provides a method for scheduling large model parameters. By setting a corresponding parameter prediction model in each layer of the transformer module of the large model, when the transformer module of this layer receives the input features, the parameter prediction model can predict the expert model required for the next layer of transformer module to perform reasoning based on the input features. Then, when it is determined that the model parameters of the expert model are not in the GPU, the model parameters can be scheduled from the CPU to the GPU to provide a data basis for the reasoning of the next layer of transformer module. Compared with the method of storing all the model parameters of the expert model in the GPU, the solution provided by the embodiment of the present application reduces the ineffective occupation of GPU memory.

[0100] It should be noted that the scheduling method for large model parameters provided in the embodiments of the present application can be applied to electronic devices, including terminals and servers; the terminals can specifically be smartphones, tablet computers, computers, personal digital assistants (PDAs), etc.; the servers can specifically be application servers or web servers. The electronic devices run the large model, parameter prediction model, cache prediction model, parameter scheduling model, cache scheduling model, etc. mentioned in the following embodiments.

[0101] Figure 1 A flow chart of a method for scheduling large model parameters provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:

[0102] Step 101: Obtain the input features of the transformer module in the current layer; the input features are the features output by the transformer module in the previous layer.

[0103] A large language model (LLM) refers to a deep learning model trained with a large amount of text data, so that the model can generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on various topics by training on huge data sets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, and to simulate the human language cognition and generation process to a certain extent. Currently commonly used large models include: GPT series, BERT series, Tongyi large model, etc. The embodiments of this application do not limit the specific types of large models.

[0104] The large model includes multiple layers of cascaded transformer modules (L layers in the embodiment of the present application), and a parameter prediction model is set in the transformer module. It can be understood that a parameter prediction model is set in each layer of transformer module, or a parameter prediction model can be set in some transformer modules. Figure 2 A large model architecture diagram provided in the embodiment of this application, such as Figure 2 As shown in the figure, each transformer module includes multiple expert models. It should be noted that transformer modules can also include shared expert models. In practical applications, some transformer modules may also include multiple expert models. The model parameters of shared expert models are always stored in GPU memory and therefore do not require scheduling.

[0105] The parameter prediction model is used to predict the target expert model required for activation during inference of the next transformer module. The input to the parameter prediction model is the input features of the transformer module in the current layer, which are the output features of the transformer module in the previous layer. It is understood that if the transformer module in the current layer is the first layer, its input features are the output features of the modules in the larger model preceding the transformer module in this layer. The parameter prediction model can be constructed using an MLP or transformer architecture.

[0106] Step 102: Analyze the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for reasoning of the next layer transformer module.

[0107] The parameter prediction model analyzes and predicts the input features of the current transformer module to obtain the target expert model required for inference in the next transformer module. The parameter prediction model can output the probability of all expert models contained in the next transformer module being activated, and then determine the target expert model from multiple expert models based on the probabilities. It should be noted that the parameter prediction model can be pre-trained before prediction.

[0108] Step 103: Determine whether the target model parameters of the target expert model need to be scheduled based on the target expert model.

[0109] In a specific implementation, the determination method is to determine whether the model parameters corresponding to the target expert model are in the GPU memory. If so, it is determined that scheduling is not required; if not, it is determined that scheduling is required. The case where the model parameters corresponding to the target expert model are not in the GPU memory may include: all model parameters corresponding to the target expert model are not in the GPU memory; and some model parameters corresponding to the target expert model are not in the GPU memory.

[0110] Step 104: If necessary, generate a parameter scheduling strategy.

[0111] The parameter scheduling policy instructs the electronic device how to schedule the target model parameters of the target expert model from the CPU memory to the GPU memory. For example, it specifies the size of the data blocks to schedule, whether to delete the model parameters stored in the GPU, and how many model parameters to delete. When generating the scheduling policy, environmental information such as the current bandwidth of the electronic device can be considered.

[0112] Step 105: Schedule the target model parameters of the target expert model from the central processing unit (CPU) memory to the graphics processing unit (GPU) memory according to the parameter scheduling strategy.

[0113] In the specific implementation process, after the parameter scheduling strategy is determined, the target model parameters are obtained from the CPU memory according to the parameter scheduling strategy, and the target model parameters are stored in the GPU memory. It should be noted that in the case where the model parameters corresponding to all target expert models are not in the GPU memory, during scheduling, the model parameters corresponding to all target expert models are scheduled from the CPU memory to the GPU memory. In the case where the model parameters corresponding to some target expert models are not in the GPU memory, only these model parameters can be scheduled from the CPU memory to the GPU memory. Moreover, after the model parameters are scheduled, the CPU memory can continue to store these model parameters, that is, the CPU memory always stores the model parameters of the full amount of expert models; or these model parameters may not be stored, and these model parameters may be scheduled to the CPU memory when they are no longer needed in the GPU memory.

[0114] The embodiment of the present application predicts the input features by using the parameter prediction model corresponding to each layer of transformer module, obtains the target expert model required for reasoning of the next layer of transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the occupancy of the GPU memory.

[0115] Based on the above embodiment, the input features are analyzed through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for reasoning of the next layer transformer module, including:

[0116] Input the input features into the parameter prediction model of the current layer transformer module to obtain the probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to represent the probability that the expert model is the expert model required for inference of the next layer transformer module;

[0117] Based on the probability value, a target expert model is selected from multiple expert models according to the expert model selection strategy;

[0118] Among them, the expert model selection strategy includes:

[0119] The number of target expert models selected from the plurality of expert models is greater than a preset number; and / or,

[0120] Starting from the maximum probability value, obtain the expert model whose sum of probability values is greater than the preset threshold.

[0121] In the specific implementation process, the probability of each expert model being activated output by the parameter prediction model can be expressed as: PE i (i=0,1,2,M-1), and Where M is the number of expert models. When selecting a target expert model from multiple expert models, the probability of each expert model being activated can be sorted and a preset threshold θ can be set. In the expert model selection strategy, the preset number can be determined based on the number of expert models activated each time in each layer of the transformer module. For example, if the number of expert models activated each time is 8, then the preset number is at least 8. In order to improve the recall rate of the model, the preset number can be set to 10, 11, etc. In addition, it is also necessary to satisfy that the sum of the probability values of the selected expert models is greater than the preset threshold θ. When selecting, the expert models can be sorted from large to small according to the probability value, and selected starting from the maximum probability value, so that the sum of the probability values of the selected expert models is just greater than the preset threshold. In other words, if one expert model is selected less, the sum of the probability values will not meet the condition of being greater than the preset threshold. That is, assuming that n expert models are selected, the selected experts must meet the following constraints: and

[0122] The embodiment of the present application selects a target expert model from multiple expert models through an expert model selection strategy. The number of selected target expert models changes dynamically according to the probability value of each expert model, which can improve the accuracy of expert model parameter scheduling.

[0123] Based on the above embodiment, a parameter scheduling strategy is generated, including:

[0124] The target model parameters, the model parameters stored in the GPU memory, and the environmental characteristics are input into the parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model; wherein the environmental characteristics include at least one of the high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, central processing unit (CPU) direct read and write bandwidth, graphics processing unit (GPU) direct read and write bandwidth, CPU occupancy, and GPU occupancy.

[0125] In the specific implementation process, the embodiment of the present application introduces a parameter scheduling model, takes the target model parameters and environmental characteristics as input, and outputs a better scheduling strategy to guide operations such as reading, writing, migration, and elimination of the target model parameters.

[0126] The target model parameters are obtained by prediction using the parameter prediction model in the above embodiment. Environmental characteristics refer to the hardware and software environmental characteristics of the electronic device, including at least one of the following:

[0127] High Bandwidth Memory (HBM) Capacity: The total capacity of HBM, which reflects how much data the system can store.

[0128] HBM occupancy: Current HBM usage, used to evaluate the remaining HBM space.

[0129] Dynamic random access memory (DRAM) capacity: The total capacity of DRAM, which also reflects storage capabilities.

[0130] DRAM occupancy: indicates the current DRAM usage and is used to evaluate the remaining DRAM space.

[0131] CPU direct read / write bandwidth: The rate at which the CPU accesses memory (including HBM and DRAM), which affects data processing speed.

[0132] GPU direct read and write bandwidth: The rate at which the GPU accesses memory is particularly important for tasks such as graphics processing and deep learning.

[0133] CPU usage: The current workload of the CPU, reflecting the utilization of its processing capacity.

[0134] GPU occupancy: The current workload of the GPU, which also reflects the utilization of its processing power.

[0135] The target model parameters refer to the model parameters of the expert model required for inference of the next layer of transformer module in the large model. The role of the model parameters stored in the GPU memory is to determine whether the target model parameters have been stored in the GPU memory, so as to determine whether the target model parameters need to be scheduled.

[0136] The target model parameters, model parameters stored in the GPU, and environmental characteristics are input into the scheduling model. The scheduling model can be based on machine learning, deep learning, or reinforcement learning. It learns and outputs the optimal parameter scheduling strategy based on the input data. Specifically, it first determines whether the target model parameters need to be scheduled and, if so, how to schedule them.

[0137] The parameter scheduling model can contain multiple processing layers, such as input layer, hidden layer and output layer, which process input data and generate output through complex network structure and algorithm.

[0138] The parameter scheduling model outputs a scheduling policy that can include the read and write order of the target model parameters, migration strategies, and elimination rules. The parameter scheduling policy aims to optimize system performance, such as reducing access latency, improving data throughput, or balancing resource usage. In the embodiments of the present application, the parameter scheduling model can take into account the current environmental characteristics, thereby allocating storage and computing resources more efficiently and reducing resource waste. It can also be applied to heterogeneous systems with different types of memory and processors to fully utilize the overall system performance.

[0139] Based on the above embodiments, an adaptive reinforcement learning framework can be used for the parameter scheduling model. Adaptive refers to a reinforcement learning scheme that can adapt to different hardware resources and different large models. In the embodiments of the present application, after a single reinforcement learning is performed on the parameter scheduling model, it can be adapted to different hardware resources. When the scheduling model is deployed on different hardware, the scheduling model can be fine-tuned to optimize model performance.

[0140] The core idea of reinforcement learning is that an agent learns optimal strategies through interaction with its environment. The agent selects an action in a given state and observes the rewards or penalties provided by the environment, adjusting its strategy to achieve greater cumulative rewards in the future. This process can be summarized as follows: the agent selects an action based on the current state, the environment returns the next state and reward based on the action, and the agent updates its strategy based on the reward.

[0141] The reinforcement learning in the embodiments of the present application is not limited to classic value-based or policy-based reinforcement learning methods, such as: Deep Q-Network, referred to as DQN; Asynchronous Advantage Actor Critic, referred to as A3C; Proximal Policy Optimization (PP0), etc.

[0142] Reinforcement learning focuses on the representation of state space, reward function, and action. The state space includes: the type of large model framework used, inferred tokens, current token, location information and storage location of the current token (including the storage location of inferred tokens and current token), HBM capacity, HBM occupancy, DRAM capacity, DRAM occupancy, CPU and GPU direct read and write bandwidth and occupancy, and the output models_needs (i.e., target model parameters) of the pre-trained parameter prediction model; it is not limited to heterogeneous storage media such as HBM and DRAM, nor is it limited to heterogeneous computing accelerators such as CPU and GPU. The state composition characteristics are expressed as follows:

[0143] S=[Token_position, Token, whole_pre_tokens, whole_DRAM, DRAM_Usage, whole_HBM, HBM_Usage, Read_bandwidth, Write_bandwidth, Read_Usage_bandwidth, Write_Usage_bandwidth; models_needs].

[0144] Among them, Token_position is the position information of the current token; Token is the current token; whole_pre_tokens is the inferred tokens; whole_DRAM is the DRAM capacity; DRAM_Usage is the DRAM occupancy; whole_HBM is the HBM capacity; HBM_Usage is the HBM occupancy; Read_bandwidth is the read bandwidth; Write_bandwidth is the write bandwidth; Read_Usage_bandwidth is the read occupancy; Write_Usage_bandwidth is the write occupancy; tokens_needs is the target KV cache.

[0145] The action is the communication or maintenance of the data required for the next layer of transformer module to infer the token. The reward function includes: the inverse of the current inference loss, HBM occupancy rate, and bandwidth occupancy rate; the reward function can be expressed as:

[0146] Reward(s,index)=-loss+HBM_Usag+Read_Usag_bandwidth+Write_Usag_bandwidth

[0147] Among them, Reward(s,index) is the reward value; loss is the loss value of the current inference; HBM_Usag is the HBM occupancy rate; Read_Usag_bandwidth is the read bandwidth occupancy rate; Write_Usag_bandwidth is the write bandwidth occupancy rate.

[0148] The action space of reinforcement learning defines the operations that can be performed, mainly including: migration operations and state-maintaining actions; migration operations include: migrating a data block from HBM to DRAM; moving a data block from DRAM back to HBM.

[0149] In the embodiment of the present application, the reinforcement learning algorithm can enable the parameter scheduling model to self-adjust and optimize according to the feedback from the environment. Through continuous learning, it can consider the occupancy and performance characteristics of various resources, thereby making a more reasonable resource allocation strategy.

[0150] In a specific implementation process, before determining whether the target model parameters of the target expert model need to be scheduled based on the target expert model, the method further includes:

[0151] The input features are analyzed through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the current layer transformer module to infer the next token.

[0152] In the specific implementation process, the parameter prediction model can not only predict the target expert model required for inference of the next layer of transformer modules, but also predict the target expert model that the current layer transformer module needs to activate when inferring the next token. Therefore, the prediction of the target expert model that needs to be activated when inferring a certain token by a certain layer of transformer modules can be predicted separately by the parameter prediction model of the current layer and the parameter prediction model of the previous layer. Therefore, the target expert model can be the union of the target expert models predicted by the parameter prediction model of the current layer and the parameter prediction model of the previous layer.

[0153] The parameter prediction model in the embodiment of the present application can not only predict the target expert model required for the next-layer transformer module reasoning, but also predict the target expert model required for the current-layer transformer module to reason about the next token. Therefore, the target expert model required for the transformer module to reason about a token is obtained by respectively predicting the parameter prediction model of the current layer and the parameter prediction model of the previous layer transformer module, thereby improving the accuracy of the prediction.

[0154] Based on the above embodiment, the capacity of GPU memory is limited and it is impossible to store data indefinitely. Therefore, when it is necessary to schedule target model parameters to GPU memory, if it is determined that the GPU memory is insufficient, deleting some model parameters from the GPU memory is an effective solution. It should be noted that the determination of whether GPU memory is insufficient can be based on pre-defined criteria. For example, if the GPU memory occupancy rate reaches 80%, it can be considered that the storage space is insufficient; or if the remaining capacity of the GPU memory is less than the size of the target model parameters to be scheduled.

[0155] When deleting some model parameters, the model parameters with the lowest access frequency can be deleted. This can be achieved by maintaining an access counter or using the LRU (Least Recently Used) cache elimination strategy. Deletion can also be performed based on importance or priority. For example, the model parameters in the GPU memory have their own priorities. When deleting, the model parameters with low priority will be deleted. It is also possible to delete according to the time of writing to the GPU memory, starting from the model parameters with the earliest writing time until the subsequent target model parameters can be written to the GPU memory. It should be noted that the model parameters of the shared expert model will not be deleted.

[0156] There are also various strategies for deleting model parameters. For example, you can delete according to a preset size, for example, deleting 5K model parameters at a time. If after deleting, the target model parameters are still not stored in the GPU memory, then delete another 5K model parameters until the target model parameters are stored in the GPU memory. You can also delete model parameters of the corresponding size from the GPU memory based on the size of the target model parameters to be scheduled.

[0157] The embodiment of the present application maintains the model parameters in the GPU memory by deleting rules, so that the model parameters of the thermal expert model are always stored in the GPU memory, reducing the number of model parameter scheduling times.

[0158] Based on the above embodiment, the method further includes:

[0159] Use the cached prediction model to predict the target key-value cache data required for the large model to infer subsequent tokens; subsequent tokens refer to tokens that have not yet been inferred by the large model;

[0160] Determine whether the target key-value cache data needs to be scheduled;

[0161] If scheduling is required, generate a cache scheduling strategy;

[0162] According to the cache scheduling strategy, the target key-value cache data is obtained from the CPU memory and stored in the GPU memory. The target key-value cache data is used to enable the large model to infer subsequent tokens. The CPU memory is used to store the full amount of key-value cache data. During the inference process, the large model reads the required key-value cache data from the GPU memory.

[0163] In the specific implementation process, due to the large computing power and bandwidth of the hardware in the decoding stage of large model reasoning, computing power resources are wasted. In order to improve the utilization rate of computing power, effectively reducing the storage-to-computation ratio is a very effective method. Researchers have successively reduced the bandwidth demand in the decoding stage by quantizing the KV cache and sparsifying the tokens. Although these methods improve TBT, they bring about a loss in the accuracy of large model reasoning. In subsequent studies, it was found that large models often do not require all tokens information during the decoding process of the inference service. Therefore, similar to the scheduling of model parameters in the above embodiments, the target key value cache data that may be used later is cached, and then these target key value cache data are scheduled to the GPU memory for subsequent reasoning use.

[0164] The cache prediction model is pre-trained, and its primary function is to predict the target KV cache required for tokens to be inferred by the large model. The prediction model may include pooling layers and fully connected layers. During training, data from commonly used KV cache sparsification schemes can be used to pre-train the small model; for example, this can generate the token sequence data required for large model inference services like Minference. As can be appreciated, since the parameters of the small model are relatively small and the inference service requires less computing power, the parameter prediction model and cache prediction model can be deployed on the electronic device's CPU or GPU, or one on the CPU and the other on the GPU. Because the amount of data output by the parameter prediction model and cache prediction model to the large model is small, their impact on bandwidth is minimal. Therefore, in scenarios with high GPU utilization, the parameter prediction model and cache prediction model can be deployed on the CPU. The deployment location of the parameter prediction model and cache prediction model can also be dynamically selected based on actual load conditions. For example, when the GPU load is low, the parameter prediction model and cache prediction model can be migrated to the GPU for execution, while switching to the CPU when the GPU load is high.

[0165] A subsequent token refers to a token that has not yet been inferred by the large model at the current moment. For example, if the large model is currently inferring the current token, the subsequent token can be the next token that the large model is about to infer, or the two tokens after the current token, etc. For ease of description, this embodiment of the application uses the subsequent token as the next token after the currently inferred token.

[0166] After obtaining the target KV cache, a determination is made as to whether the target KV cache needs to be scheduled. The primary determination method is to determine whether the target KV cache is contained in GPU memory. If so, scheduling is determined not to be necessary; otherwise, scheduling is determined to be necessary. If only a portion of the target KV cache is contained in GPU memory, it is also considered that the target KV cache is not contained and scheduling is required. However, during the subsequent scheduling process, only the portion of the target KV cache not contained in GPU memory is stored in GPU memory. For example, if the target KV cache includes KV caches corresponding to tokens 1-10, 30-50, and 100-110, and the GPU memory currently stores the KV caches corresponding to tokens 1-10 and 30-50, but not the KV cache corresponding to tokens 100-110, scheduling is determined to be necessary. During scheduling, the KV caches corresponding to tokens 100-110 are stored in GPU memory.

[0167] It is understood that the GPU memory is the storage space of the target KV cache that needs to be read by the large model during the inference process. Specifically, it can be the HBM in the GPU, or it can be other storage space in the GPU. The embodiments of the present application do not specifically limit this. In addition, the cache prediction model can output the index value of the target KV cache, and use the index value to search the GPU memory to see if there is a KV cache corresponding to the same index value to determine whether scheduling is required. Moreover, when scheduling is required, the corresponding KV cache can also be obtained from the CPU memory based on the index value and then stored in the GPU memory.

[0168] The cache scheduling policy instructs the electronic device how to schedule the target KV cache from CPU memory to GPU memory. For example, it specifies the data block size for scheduling, whether to delete the KV cache in GPU memory, and how much KV cache to delete. When generating the scheduling policy, environmental information such as the current bandwidth of the electronic device can be considered.

[0169] In the case where the KV cache in the GPU needs to be deleted, the deletion strategy can refer to the deletion strategy for the model parameters in the GPU in the above embodiment, which will not be repeated here.

[0170] After the cache scheduling strategy is determined, the target KV cache is obtained from the CPU memory according to the cache scheduling strategy, and the target KV cache is stored in the GPU memory.

[0171] The embodiment of the present application predicts the target key-value cache data required for the subsequent token reasoning of the large model through a prediction model, and performs scheduling judgment, scheduling strategy generation, and data migration operations on the target KV cache, thereby reducing frequent access to the first storage space. In addition, only the required target key-value cache data is transmitted each time instead of the full amount of key-value cache data, which greatly reduces the bandwidth resource occupation caused by transmitting key-value cache data.

[0172] Based on the above embodiment, the cache prediction model is used to predict the target key value cache data required for the large model to infer subsequent tokens, including:

[0173] Input the inferred tokens, current token, and the location information of the current token into the cache prediction model to obtain the target key-value cache data required for subsequent token inference output by the cache prediction model; among them, the current token refers to the token being inferred by the large model.

[0174] In the specific implementation process, in the process of large model reasoning, in order to improve the reasoning efficiency, KV cache is usually used to store the calculated hidden states or intermediate results to avoid repeated calculations.

[0175] Inferred tokens refer to tokens that have been inferred by the large model. Specifically, they can be represented in the form of token sequences. The hidden states or intermediate results of these tokens have been stored in the KV cache.

[0176] The current token refers to the token that the large model is inferring. The location information of the current token refers to the position of the token that the large model is inferring in the inference service sequence. It can be represented by the index value index, which is used to predict the sequential relationship of tokens understood by the model.

[0177] The inferred tokens, the current token, and the location information of the current token are combined into a complete input sequence and input into the prediction model. The prediction model generates the target KV cache required by the large model to infer the next token based on the input content.

[0178] The embodiment of the present application uses a prediction model to analyze the inferred tokens, the current token, and the location information of the current token, so as to capture context information and more accurately predict the target key-value cache data required for subsequent tokens.

[0179] Based on the above embodiment, a cache scheduling strategy is generated, including:

[0180] The target key-value cache data, the key-value cache data stored in the GPU memory, and the environmental characteristics are input into the cache scheduling model to obtain a cache scheduling policy output by the cache scheduling model; wherein the environmental characteristics include at least one of high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, CPU direct read and write bandwidth, GPU direct read and write bandwidth, CPU occupancy, and GPU occupancy.

[0181] In the specific implementation process, the embodiment of the present application introduces a cache scheduling model, takes the target KV cache and environmental characteristics as input, and outputs a better cache scheduling strategy to guide the reading, writing, migration and elimination operations of the KV cache.

[0182] Among them, the target KV cache is obtained by prediction through the prediction model in the above embodiment. Environmental characteristics refer to the hardware and software environmental characteristics of the electronic device, including at least one of the following: high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, CPU direct read and write bandwidth, GPU direct read and write bandwidth, CPU occupancy and GPU occupancy. For the relevant explanations of the above-mentioned environmental characteristics, please refer to the above-mentioned embodiments and will not be repeated here. The method for generating a cache scheduling strategy based on the above-mentioned environmental characteristics, target key-value cache data and key-value cache data stored in the GPU memory is similar to the method for generating a parameter scheduling strategy and will not be repeated here.

[0183] In the embodiment of the present application, the scheduling model can take into account the current environmental characteristics, so that storage and computing resources can be allocated more efficiently, reducing resource waste; and can be applied to heterogeneous systems with different types of memory and processors to fully utilize the overall performance of the system.

[0184] Figure 3 The large model reasoning architecture diagram provided in the embodiment of this application is as follows: Figure 3 As shown. The large model is deployed on an electronic device, which includes a CPU and a GPU. The CPU includes a CPU computing core and DRAM. Among them, the DRAM stores the full amount of KV cache, and can also store parameter prediction models, parameter scheduling models, cache prediction models and cache scheduling models. The GPU includes a GPU computing core and HBM, and the HBM stores the weights of the large model LLM, thermal model parameters and hot KV cache. It can be understood that the thermal model parameters refer to the expert model parameters frequently used by the large model during the reasoning process. Hot KV cache refers to the KV cache frequently used in the large model reasoning process, as well as the KV cache required for reasoning subsequent tokens.

[0185] The scheduling of model parameters and the scheduling of KV cache can adopt the above embodiments, which will not be described in detail here.

[0186] Figure 4 A flow chart of a large model reasoning method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method includes:

[0187] Step 401: Input the inferred tokens into the large model; the large model includes multiple layers of transformer modules, each layer of transformer modules includes multiple expert models;

[0188] Step 402: Determine an activated expert model from multiple expert models based on the input features of each transformer module;

[0189] Step 403: When the target model parameters stored in the GPU memory include activation model parameters corresponding to all activated expert models, infer the input features based on the activation model parameters corresponding to the activated expert models to obtain output features;

[0190] The target model parameters are obtained using the method of the above embodiment.

[0191] In the specific implementation process, the inferred tokens are input into the large model, and the inference is performed layer by layer through the transformer modules of each layer of the large model. During the inference process of each layer of the transformer module, since each layer of the transformer module includes multiple expert models, the router will select the expert model that needs to be activated based on the input features, which is referred to as the activated expert model in the embodiment of the present application. When analyzing and inferring the input features through the activated expert model, it is necessary to obtain the model parameters of the activated expert model from the GPU memory. If the GPU memory includes the model parameters of all activated expert models, then inference can be performed after obtaining the model parameters. The result obtained after the inference of the transformer module of the current layer is input into the transformer module of the next layer, and then inference is performed again according to the above method to finally obtain the output features of the large model output.

[0192] The embodiment of the present application predicts the input features by using the parameter prediction model corresponding to each layer of transformer module, obtains the target expert model required for reasoning of the next layer of transformer module, and schedules the target model parameters corresponding to the target expert model from the CPU to the GPU, thereby reducing the occupancy of the GPU memory.

[0193] Based on the above embodiment, since the model parameters stored in GPU memory are obtained based on predictions from the parameter prediction model, and the parameter prediction model's prediction accuracy is not 100%, during the inference process of a large model, the model parameters of the required activated expert model may not be in GPU memory, while the model parameters of all expert models are always stored in CPU memory. To address this situation, input features can be inferred based on some of the activated model parameters stored in GPU memory and other activated model parameters stored in CPU memory to obtain output features.

[0194] In the specific implementation process, for the activation model parameters of the partial activation expert model stored in GPU memory, these activation expert models can continue to be inferred on the GPU. For the activation model parameters not stored in GPU memory, the input features can be input into CPU memory, and inference can be performed using the activation expert models corresponding to the remaining activation model parameters in the CPU. Finally, the output of the activation expert model in the GPU and the output of the activation expert model in the CPU are integrated to obtain the final result.

[0195] In an embodiment of the present application, the target expert model predicted by the parameter prediction model may be inaccurate, that is, the GPU may not contain all the model parameters of the activated expert model required for reasoning of a certain transformer layer of the large model. In this case, the GPU and CPU can be connected to jointly perform reasoning without the need to schedule the remaining model parameters of the activated expert model from the CPU to the GPU again, thereby improving the efficiency of model reasoning.

[0196] Based on the above embodiment, the method further includes:

[0197] Obtain target key-value cache data of the token to be inferred through the large model; wherein the target key-value cache data is obtained by using the scheduling method of the large model parameters described in the above embodiment;

[0198] Correspondingly, the partial activation model parameters stored in GPU memory include:

[0199] Reasoning about input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

[0200] In the specific implementation process, during the inference process, the large model not only needs to activate the model parameters of the expert model, but also needs to cache data based on the target key value. Therefore, the large model can infer the input features based on the model parameters corresponding to the activated expert model and the target key value cache data stored in the GPU memory.

[0201] The embodiment of the present application infers input features based on the model parameters and target key-value cache data corresponding to the activated expert model, wherein the model parameters and target key-value cache data corresponding to the activated expert model are obtained through targeted scheduling from the CPU, thereby reducing frequent access to the first storage space and improving reasoning efficiency.

[0202] In addition, large-model inference can be performed asynchronously with parameter and cache prediction and scheduling. For example, while the large model is inferencing the current token, the cache prediction model can predict the target KV cache required for inferencing the next token. If the target KV cache is already stored in GPU memory, no scheduling is required. If the target KV cache is not in GPU memory, the cache scheduling model schedules the target KV cache from CPU memory to GPU memory. When the large model infers the next token, it can directly read the target KV cache from GPU memory. At this point, the prediction model can predict the target KV cache required for inferencing the next token, and so on. Furthermore, when inferring a transformer module in a certain layer of the large model, the parameter prediction module of that layer can predict the target model parameters required for inferencing the transformer module in the next layer, as well as the target model parameters required for inferring the next token in that transformer module. If the target model parameters are not in GPU memory, they are scheduled from CPU memory to GPU memory. Through asynchronous execution, when a large model infers a token, it can directly read the target KV cache and activation model parameters required for token inference from the GPU. This saves the time of reading the full KV cache and activation model parameters from the CPU, improving inference efficiency; it also reduces the number of KV cache and activation model parameter reads, reducing bandwidth usage.

[0203] Figure 5 This is a schematic diagram of a large model parameter scheduling device provided in an embodiment of the present application. The device can be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 1 The method embodiment corresponds to the embodiment that can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the description above. To avoid repetition, detailed description is omitted here. The device includes: a feature acquisition module 501, an expert model prediction module 502, a judgment module 503, a scheduling strategy generation module 504 and a scheduling module 505, wherein:

[0204] The feature acquisition module 501 is used to obtain the input features of the current layer transformer module; the input features are the features output by the previous layer transformer module;

[0205] The expert model prediction module 502 is used to analyze the input features through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the next layer transformer module reasoning;

[0206] The judgment module 503 is used to judge whether the target model parameters of the target expert model need to be scheduled based on the target expert model;

[0207] The scheduling strategy generation module 504 is used to generate a parameter scheduling strategy when the target model parameters need to be scheduled;

[0208] The scheduling module 505 is used to schedule the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy.

[0209] Based on the above embodiment, the judgment module 503 is specifically configured to:

[0210] Determine whether the target model parameters of the target expert model are stored in the GPU memory;

[0211] If the target model parameters are not stored in the GPU memory, it is determined that scheduling is required; otherwise, scheduling is not required.

[0212] Based on the above embodiment, the expert model prediction module 502 is specifically used to:

[0213] Inputting the input features into the parameter prediction model of the transformer module of the current layer, obtaining a probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to represent the probability that the expert model is the expert model required for inference of the transformer module of the next layer;

[0214] Based on the probability value, selecting the target expert model from multiple expert models according to an expert model selection strategy;

[0215] The expert model selection strategy includes:

[0216] The number of the target expert models selected from the plurality of expert models is greater than a preset number; and / or,

[0217] Starting from the maximum probability value, obtain the expert model whose sum of probability values is greater than the preset threshold.

[0218] Based on the above embodiment, the scheduling strategy generation module 504 is specifically used to:

[0219] The target model parameters, the model parameters stored in the GPU memory, and the environmental characteristics are input into a parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model; wherein the environmental characteristics include at least one of high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, central processing unit (CPU) direct read and write bandwidth, graphics processing unit (GPU) direct read and write bandwidth, CPU occupancy, and GPU occupancy.

[0220] Based on the above embodiment, the parameter scheduling model is obtained through training based on a reinforcement learning algorithm.

[0221] Based on the above embodiment, the expert model prediction module 502 is further configured to:

[0222] The input features are analyzed through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the current layer transformer module to infer the next token.

[0223] Based on the above embodiment, the device further includes a deletion module, which is used to:

[0224] If the GPU memory is insufficient, some model parameters are deleted from the GPU memory based on the deletion rule.

[0225] Based on the above embodiment, the device further includes a cache prefetching module, which is used to:

[0226] Utilize the cache prediction model to predict the target key-value cache data required for the large model to infer subsequent tokens; the subsequent tokens are tokens that have not yet been inferred by the large model;

[0227] Determine whether the target key-value cache data needs to be scheduled;

[0228] If scheduling is required, generate a cache scheduling strategy;

[0229] The target key-value cache data is obtained from the CPU memory according to the cache scheduling strategy, and the target key-value cache data is stored in the GPU memory; wherein, the target key-value cache data is used to enable the large model to infer the subsequent token; the CPU memory is used to store the full amount of key-value cache data; during the inference process, the large model reads the required key-value cache data from the GPU memory.

[0230] Based on the above embodiment, the cache prefetch module is specifically used to:

[0231] The inferred tokens, the current token, and the location information of the current token are input into the cache prediction model to obtain the target key-value cache data required for the subsequent token inference output by the cache prediction model; wherein the current token refers to the token being inferred by the large model.

[0232] Based on the above embodiment, the cache prefetch module is specifically used to:

[0233] The target key-value cache data, the key-value cache data stored in the GPU memory, and the environmental characteristics are input into a cache scheduling model to obtain a cache scheduling policy output by the cache scheduling model; wherein the environmental characteristics include at least one of high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, CPU direct read and write bandwidth, GPU direct read and write bandwidth, CPU occupancy, and GPU occupancy.

[0234] Figure 6 This is a schematic diagram of a large model reasoning device provided in an embodiment of the present application. The device can be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 4 The method embodiment corresponds to the embodiment that can be executed Figure 4 The various steps involved in the method embodiment and the specific functions of the device can be found in the description above. To avoid repetition, detailed description is omitted here. The device includes: an input module 601, an expert model determination module 602 and an inference module 603, wherein:

[0235] The input module 601 is used to input the inferred tokens into the large model; the large model includes multiple layers of transformer modules, and each layer of transformer modules includes multiple expert models;

[0236] The expert model determination module 602 is used to determine an activated expert model from multiple expert models based on the input features of the transformer module at each layer;

[0237] The inference module 603 is configured to, when the target model parameters stored in the GPU memory include all the activation model parameters corresponding to the activated expert model, infer the input features based on the activation model parameters corresponding to the activated expert model to obtain output features;

[0238] The target model parameters are obtained using the method described in the above embodiment.

[0239] Based on the above embodiment, the reasoning module 603 is further configured to:

[0240] In case that the target model parameters stored in the GPU memory include some activation model parameters corresponding to the activated expert model, reasoning on the input features based on the some activation model parameters stored in the GPU memory and based on other activation model parameters stored in the CPU memory;

[0241] The CPU memory stores model parameters corresponding to all expert models.

[0242] Based on the above embodiment, the cache data acquisition module is used to:

[0243] Obtain target key-value cache data of the token to be inferred through the large model; wherein the target key-value cache data is obtained by using the scheduling method of the large model parameters described in the above embodiment;

[0244] Accordingly, the reasoning module 603 is specifically used to:

[0245] Reasoning about input features based on the model parameters corresponding to the activated expert model and the target key-value cache data.

[0246] Based on the above embodiment, the inference process of the large model and the scheduling process of the target model parameters and the scheduling process of the target key-value cache data are executed asynchronously.

[0247] Figure 7 The physical structure diagram of the electronic device provided in the embodiment of the present application is as follows: Figure 7 As shown, the electronic device includes: a processor (processor) 701, a memory (memory) 702 and a bus 703; wherein:

[0248] The processor 701 and the memory 702 communicate with each other via the bus 703;

[0249] The processor 701 is used to call the program instructions in the memory 702 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the input features of the transformer module of the current layer; the input features are the features output by the transformer module of the previous layer; analyzing the input features through the parameter prediction model corresponding to the transformer module of the current layer to obtain the target expert model required for the inference of the transformer module of the next layer; judging whether the target model parameters of the target expert model need to be scheduled based on the target expert model; if necessary, generating a parameter scheduling strategy; scheduling the target model parameters of the target expert model from the central processing unit CPU memory to the image processing unit GPU memory according to the parameter scheduling strategy. Or,

[0250] The inferred tokens are input into the large model; the large model includes multiple layers of transformer modules, and each layer of transformer modules includes multiple expert models; the activated expert model is determined from the multiple expert models through the input features of each layer of transformer modules; for the case where the target model parameters stored in the GPU memory include the activation model parameters corresponding to all activated expert models, the input features are inferred based on the activation model parameters corresponding to the activated expert models to obtain output features; wherein the target model parameters are obtained using the method of the above embodiment.

[0251] The processor 701 can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 701 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0252] The memory 702 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0253] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the input features of the transformer module of the current layer; the input features are the features output by the transformer module of the previous layer; analyzing the input features through the parameter prediction model corresponding to the transformer module of the current layer to obtain the target expert model required for the inference of the transformer module of the next layer; judging whether the target model parameters of the target expert model need to be scheduled based on the target expert model; if necessary, generating a parameter scheduling strategy; scheduling the target model parameters of the target expert model from the central processing unit CPU memory to the image processing unit GPU memory according to the parameter scheduling strategy. Or,

[0254] The inferred tokens are input into the large model; the large model includes multiple layers of transformer modules, and each layer of transformer modules includes multiple expert models; the activated expert model is determined from the multiple expert models through the input features of each layer of transformer modules; for the case where the target model parameters stored in the GPU memory include the activation model parameters corresponding to all activated expert models, the input features are inferred based on the activation model parameters corresponding to the activated expert models to obtain output features; wherein the target model parameters are obtained using the method of the above embodiment.

[0255] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the input features of the transformer module of the current layer; the input features are the features output by the transformer module of the previous layer; analyzing the input features through the parameter prediction model corresponding to the transformer module of the current layer to obtain the target expert model required for the inference of the transformer module of the next layer; judging whether the target model parameters of the target expert model need to be scheduled based on the target expert model; if necessary, generating a parameter scheduling strategy; scheduling the target model parameters of the target expert model from the central processing unit CPU memory to the image processing unit GPU memory according to the parameter scheduling strategy. Or,

[0256] The inferred tokens are input into the large model; the large model includes multiple layers of transformer modules, and each layer of transformer modules includes multiple expert models; the activated expert model is determined from the multiple expert models through the input features of each layer of transformer modules; for the case where the target model parameters stored in the GPU memory include the activation model parameters corresponding to all activated expert models, the input features are inferred based on the activation model parameters corresponding to the activated expert models to obtain output features; wherein the target model parameters are obtained using the method of the above embodiment.

[0257] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0258] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0260] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0261] The above are merely examples of the present application and are not intended to limit the scope of protection of the present application. Those skilled in the art will appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for scheduling large model parameters, characterized in that: The large model includes multiple layers of transformer modules, a parameter prediction model is provided for each transformer module, and the method includes: Get the input features of the transformer module in the current layer; the input features are the features output by the transformer module in the previous layer; Analyze the input features through the parameter prediction model corresponding to the transformer module of the current layer to obtain the target expert model required for reasoning of the transformer module of the next layer; Determining whether it is necessary to schedule target model parameters of the target expert model based on the target expert model; If necessary, generate parameter scheduling strategy; The target model parameters of the target expert model are scheduled from the central processing unit (CPU) memory to the graphics processing unit (GPU) memory according to the parameter scheduling strategy.

2. The method according to claim 1, characterized in that The determining whether it is necessary to schedule the target model parameters of the target expert model based on the target expert model includes: Determine whether the target model parameters of the target expert model are stored in the GPU memory; If the target model parameters are not stored in the GPU memory, it is determined that scheduling is required; otherwise, scheduling is not required.

3. The method according to claim 1, characterized in that The step of analyzing the input features by using the parameter prediction model corresponding to the transformer module of the current layer to obtain the target expert model required for reasoning of the transformer module of the next layer includes: Inputting the input features into the parameter prediction model of the transformer module of the current layer, obtaining a probability value corresponding to each expert model output by the parameter prediction model; the probability value is used to represent the probability that the expert model is the expert model required for inference of the transformer module of the next layer; Based on the probability value, the target expert model is selected from multiple expert models according to an expert model selection strategy.

4. The method according to claim 3, characterized in that The expert model selection strategy includes: The number of the target expert models selected from the plurality of expert models is greater than a preset number; and / or, Starting from the maximum probability value, obtain the expert model whose sum of probability values is greater than the preset threshold.

5. The method according to claim 1, wherein The generating parameter scheduling strategy includes: The target model parameters, the model parameters stored in the GPU memory, and the environmental characteristics are input into a parameter scheduling model to obtain the parameter scheduling strategy output by the parameter scheduling model; wherein the environmental characteristics include at least one of high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, central processing unit (CPU) direct read and write bandwidth, graphics processing unit (GPU) direct read and write bandwidth, CPU occupancy, and GPU occupancy.

6. The method according to claim 5, characterized in that The parameter scheduling model is obtained through training based on a reinforcement learning algorithm.

7. The method according to claim 1, characterized in that Before determining whether target model parameters of the target expert model need to be scheduled based on the target expert model, the method further includes: The input features are analyzed through the parameter prediction model corresponding to the current layer transformer module to obtain the target expert model required for the current layer transformer module to infer the next token.

8. The method according to claim 1, characterized in that Before scheduling the target model parameters of the target expert model from the central processing unit (CPU) memory to the graphics processing unit (GPU) memory according to the parameter scheduling strategy, the method further includes: If the GPU memory is insufficient, some model parameters are deleted from the GPU memory based on the deletion rule.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Utilize the cache prediction model to predict the target key-value cache data required for the large model to infer subsequent tokens; the subsequent tokens are tokens that have not yet been inferred by the large model; Determine whether the target key-value cache data needs to be scheduled; If scheduling is required, generate a cache scheduling strategy; The target key-value cache data is obtained from the CPU memory according to the cache scheduling strategy, and the target key-value cache data is stored in the GPU memory; wherein, the target key-value cache data is used to enable the large model to infer the subsequent token; the CPU memory is used to store the full amount of key-value cache data; during the inference process, the large model reads the required key-value cache data from the GPU memory.

10. The method according to claim 9, characterized in that The target key value cache data required for predicting subsequent tokens by using the cache prediction model includes: The inferred tokens, the current token, and the location information of the current token are input into the cache prediction model to obtain the target key-value cache data required for the subsequent token inference output by the cache prediction model; wherein the current token refers to the token being inferred by the large model.

11. The method according to claim 9, characterized in that Generating a cache scheduling strategy includes: The target key-value cache data, the key-value cache data stored in the GPU memory, and the environmental characteristics are input into a cache scheduling model to obtain a cache scheduling policy output by the cache scheduling model; wherein the environmental characteristics include at least one of high-bandwidth memory HBM capacity, HBM occupancy, dynamic random access memory DRAM capacity, DRAM occupancy, CPU direct read and write bandwidth, GPU direct read and write bandwidth, CPU occupancy, and GPU occupancy.

12. A large model reasoning method, characterized in that: The large model includes multiple layers of transformer modules, each layer of which is equipped with a parameter prediction model, including: Input the inferred tokens into a large model; the large model includes multiple layers of transformer modules, each layer of transformer modules includes multiple expert models; Determine an activated expert model from multiple expert models through the input features of the transformer module in each layer; In the case where the target model parameters stored in the GPU memory include all the activation model parameters corresponding to the activated expert model, inferring the input features based on the activation model parameters corresponding to the activated expert model to obtain output features; The target model parameters are obtained by using the method according to any one of claims 1 to 8.

13. The method according to claim 12, characterized in that The method further comprises: In case that the target model parameters stored in the GPU memory include some activation model parameters corresponding to the activated expert model, reasoning on the input features based on the some activation model parameters stored in the GPU memory and based on other activation model parameters stored in the CPU memory; The CPU memory stores model parameters corresponding to all expert models.

14. The method according to claim 12, characterized in that The method further comprises: Obtain target key-value cache data of the token to be inferred through the large model; wherein the target key-value cache data is obtained by using the scheduling method of the large model parameters according to any one of claims 9 to 11; Accordingly, reasoning the input features based on the activation model parameters corresponding to the activated expert model includes: The input features are inferred based on model parameters corresponding to the activated expert model and the target key-value cache data.

15. The method according to claim 12, characterized in that The inference process of the large model, the scheduling process of the target model parameters, and the scheduling process of the target key-value cache data are executed asynchronously.

16. A scheduling device for large model parameters, characterized in that: The large model includes multiple layers of transformer modules and a parameter prediction model corresponding to each layer of the transformer modules. The device includes: A feature acquisition module is used to obtain the input features of the transformer module in the current layer; the input features are the features output by the transformer module in the previous layer; An expert model prediction module is used to analyze the input features through the parameter prediction model corresponding to the transformer module of the current layer to obtain the target expert model required for reasoning of the transformer module of the next layer; A judgment module, configured to judge whether target model parameters of the target expert model need to be scheduled based on the target expert model; A scheduling strategy generation module, used for generating a parameter scheduling strategy when the target model parameters need to be scheduled; A scheduling module is used to schedule the target model parameters of the target expert model from the CPU memory to the GPU memory according to the parameter scheduling strategy.

17. A large model reasoning device, characterized in that: include: An input module, configured to input inferred tokens into a large model comprising multiple layers of transformer modules, each layer of which comprises multiple expert models; An expert model determination module, configured to determine an activated expert model from a plurality of expert models based on the input features of the transformer module at each layer; An inference module, configured to, in a case where the target model parameters stored in the GPU memory include all the activation model parameters corresponding to the activation expert model, infer the input features based on the activation model parameters corresponding to the activation expert model to obtain output features; The target model parameters are obtained by using the method according to any one of claims 1 to 8.

18. An electronic device, characterized in that: include: A processor, memory, and bus, where: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 15.

19. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, enable the computer to perform the method according to any one of claims 1 to 15.

20. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 15 is executed.

Images