A model inference method and system

Abstract

A model inference method and system are disclosed, relating to the field of AI technology. The model inference system obtains a first inference request. A first accelerator processes the first inference request according to a first hot weight to obtain a first inference result, and a second accelerator processes the first inference request according to a second hot weight to obtain a second inference result, and then the first accelerator obtains an output result according to the first inference result and the second inference result. The activation frequency of the first hot weight is greater than the activation frequency of a cold weight in the model, and the first hot weight is the same as the second hot weight. The same hot weight is deployed in the first accelerator and the second accelerator, and then the first accelerator and the second accelerator can obtain the hot weight from their local positions to perform model inference when performing model inference. Since the speed of the accelerator in obtaining the hot weight from its local position is greater than the speed of the accelerator in obtaining the weight from other positions (such as a processor), the efficiency of model inference can be improved.

Description

[0001] This application is a divisional application of Chinese patent application filed on September 12, 2024, with application number 202411280640.2 and invention title "A Model Reasoning Method and System". Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model reasoning method and system. Background Technology

[0003] Model inference refers to the process of using a trained model to process input data and output a result. In this process, the computing device retrieves the model's weights from memory to process the input data. In a single-machine scenario, some weights are typically deployed on an accelerator card, while others are deployed on the processor. When the accelerator card processes the input data, it needs to retrieve the weights from the processor. However, communication between the accelerator card and the processor is often via a high-speed serial computer expansion bus (Peripheral Component Interconnect Express, PCIe). PCIe has limited bandwidth, and the accelerator card's ability to retrieve weights is slow, resulting in low efficiency for model inference. Summary of the Invention

[0004] This application provides a model inference method and system to solve the problem that due to the limited PCIe bandwidth, the accelerator card is slow to obtain weights from the processor, resulting in low efficiency of model inference.

[0005] Firstly, this application provides a model inference method. This model inference method can be applied to a computer system or a computing device implementing the model inference method within that computer system, such as a model inference system, where the computing device includes an accelerator. In one possible example, the model inference system includes multiple accelerators, including a first accelerator and a second accelerator. The first accelerator is configured with first hot weights of the model, and the second accelerator is configured with second hot weights of the model. The model inference method provided in this aspect includes: the model inference system acquiring a first inference request; the first accelerator processing the first inference request according to the first hot weights to obtain a first inference result, and the second accelerator processing the first inference request according to the second hot weights to obtain a second inference result; and the first accelerator obtaining an output result based on the first and second inference results. Wherein, the activation frequency of the first hot weights is greater than the activation frequency of the cold weights in the model, and the first hot weights are the same as the second hot weights.

[0006] In this application, both the first and second accelerators are equipped with the same hot weights. Therefore, when performing model inference, the first and second accelerators can obtain the hot weights locally for model inference. Since the accelerators (first and second) obtain hot weights locally at a faster rate than they obtain weights from other locations (such as the processor), the efficiency of model inference can be improved. Furthermore, since the activation frequency of hot weights is greater than the activation frequency of cold weights in the model, when the first and second accelerators process the first inference request, they can obtain most of the weights required to execute the first inference request locally. This reduces the amount of data the first and second accelerators need to obtain weights from other locations. Furthermore, since the accelerators obtain hot weights locally at a faster rate than they obtain weights from other locations, the efficiency of model inference can be further improved.

[0007] In one possible scenario, the aforementioned model could be a large language model (LLM), a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or the like.

[0008] In one possible scenario, the weights corresponding to a network layer in the above model include the first hot weights.

[0009] In another possible scenario, the weights corresponding to multiple network layers in the above model include the first hot weights.

[0010] In one possible example, the model inference system obtains a first inference request, including: a first accelerator obtaining a first inference request.

[0011] In another possible example, the model inference system also includes a processor that acquires a first inference request, including: the processor acquiring the first inference request.

[0012] For example, the processor obtains the first inference request through the transceiver.

[0013] In one possible implementation, the first accelerator and the second accelerator are of the same type.

[0014] In one possible example, both the first and second accelerators are processors used to instruct model inference.

[0015] For example, both the first and second accelerators are graphics processing units (GPUs).

[0016] For example, both the first and second accelerators are neural network processing units (NPUs).

[0017] For example, both the first and second accelerators are tensor processing units (TPUs).

[0018] For example, the first accelerator and the second accelerator are both field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or data processing units (DPUs).

[0019] In one possible implementation, the aforementioned accelerators further include a third accelerator, which is of a different type from the first and second accelerators. The third accelerator is configured with third hot weights for the model, which are different from the first hot weights. Furthermore, the aforementioned model inference method further includes: the third accelerator processing the first inference request according to the third hot weights to obtain a third inference result; and the first accelerator obtaining an output result based on the first and second inference results, including: the first accelerator obtaining an output result based on the first, second, and third inference results.

[0020] In this application, since the model inference system includes different types of accelerators and these different types of accelerators store different hot weights, model inference can be realized through a heterogeneous model inference system. The heterogeneous model inference system can flexibly cope with different inference requirements and loads, thereby improving the efficiency of model inference.

[0021] In one possible scenario, the first and second accelerators are of type GPU, and the third accelerator is of type NPU.

[0022] In another possible scenario, the first and second accelerators are of type GPU, and the third accelerator is of type TPU.

[0023] In one possible example, the first hot weights stored in the first accelerator are the weights of the first network layer in the model, and the third hot weights stored in the third accelerator are the weights of the second network layer in the model. The first network layer and the second network layer are different network layers in the model.

[0024] In one possible implementation, the first accelerator processes the first inference request according to the first hot weights to obtain the first inference result, including: the first accelerator processes the first inference request using a first sub-hot weight from the first hot weights to obtain the first inference result. The first sub-hot weight is the weight that the model inference system predicts will be used when processing the first inference request using the first hot weights, as determined by the flying line predictor.

[0025] In this application, the first accelerator only needs to activate the first sub-hot weight when processing the first inference request, that is, the first accelerator only needs to acquire the first sub-hot weight, which reduces the amount of data acquired by the first accelerator for weight, thereby reducing the time required to acquire weight, and thus improving the efficiency of the first accelerator in performing model inference.

[0026] Similarly, in one possible implementation, the second accelerator processes the first inference request according to the second hot weights to obtain a second inference result, including: the second accelerator uses the second sub-hot weights in the second hot weights to process the first inference request and obtain the second inference result. The second sub-hot weights are the weights that the model inference system will use when the first hot weights process the first inference request, as predicted by the flying line predictor.

[0027] In this application, the second accelerator only needs to activate the second sub-hot weights when processing the first inference request. That is, the second accelerator only needs to acquire the second sub-hot weights, which reduces the amount of data the second accelerator needs to acquire weights, thereby reducing the time required to acquire weights and improving the efficiency of the second accelerator in performing model inference.

[0028] In one possible implementation, the first accelerator is further configured with a first cold weight, and the second accelerator is further configured with a second cold weight. The first accelerator processes the first inference request based on the first hot weight to obtain a first inference result, including: the first accelerator processes the first inference request based on the first hot weight and the first cold weight to obtain the first inference result. The second accelerator processes the first inference request based on the second hot weight to obtain a second inference result, including: the second accelerator processes the first inference request based on the second hot weight and the second cold weight to obtain the second inference result.

[0029] In this application, since the first accelerator is configured with a first cold weight and the second accelerator is configured with a second cold weight, the amount of data that the first or second accelerator can obtain from other devices in the model inference system can be reduced, thereby reducing the time required to obtain the weight and improving the efficiency of the model inference system in performing model inference.

[0030] In one possible scenario, the distance between the storage region of the first hot weight and the acceleration chip in the first accelerator is less than or equal to the distance between the storage region of the first cold weight and the aforementioned acceleration chip, thereby the memory access efficiency of the acceleration chip for the first hot weight is greater than or equal to the memory access efficiency of the acceleration chip for the first cold weight.

[0031] In one possible example, the first hot weights are stored in the first-level cache, second-level cache, or third-level cache of the first accelerator. In one possible implementation, the first cold weights are predicted by the model inference system using a flyline predictor to determine the weights required when processing the first inference request. These required weights include: the first sub-hot weight in the first hot weights, the first sub-cold weight in the first cold weights, the second sub-hot weight in the second hot weights, and the second sub-cold weight in the second cold weights. The first accelerator processes the first inference request based on the first hot weights and the first cold weights to obtain a first inference result, including: the first accelerator processes the first inference request based on the first sub-hot weights and the first sub-cold weights to obtain a first inference result. The second accelerator processes the first inference request based on the second hot weights and the second cold weights to obtain a second inference result, including: the second accelerator processes the first inference request based on the second sub-hot weights and the second sub-cold weights to obtain a second inference result.

[0032] In this application, when processing the first inference request, the first accelerator only needs to activate the first sub-hot weights and the first sub-cold weights. That is, the first accelerator only needs to acquire the first sub-hot weights and the first sub-cold weights, which reduces the amount of data the first accelerator needs to acquire weights (from the amount of data of the first hot weights and the first sub-cold weights to the amount of data of the first sub-hot weights and the first sub-cold weights), thereby reducing the time required for the first accelerator to acquire weights and thus improving the efficiency of model inference.

[0033] Similarly, when processing the first inference request, the second accelerator only needs to activate the second sub-hot weights and the second sub-cold weights. This means the second accelerator only needs to acquire the second sub-hot weights and the second sub-cold weights, reducing the amount of data required for weight acquisition (from the amount of data for the second hot weights and the second sub-cold weights to the amount of data for the second sub-hot weights and the second sub-cold weights). This reduces the time required for the second accelerator to acquire weights, thereby improving the efficiency of model inference. In summary, this improves the efficiency of the model inference system in performing model inference.

[0034] In one possible implementation, the difference between the sum of the data amounts of the first sub-hot weight and the first sub-cold weight and the sum of the data amounts of the second sub-hot weight and the second sub-cold weight is less than or equal to a first threshold.

[0035] In this application, since the difference between the sum of the data volume of the first sub-hot weight and the first sub-cold weight and the sum of the data volume of the second sub-hot weight and the second sub-cold weight is less than or equal to a first threshold, the difference in hardware resources (computing resources, storage resources, network resources) required by each accelerator (first accelerator and second accelerator) when processing the first inference request is within the first threshold. In other words, each accelerator is load-balanced when processing the first inference request, thereby improving the efficiency of the model inference system in performing model inference.

[0036] In one possible implementation, the first cold weights include weight combinations with a co-occurrence rate greater than or equal to a second threshold, and the weight combinations are stored in the same storage unit in the first accelerator.

[0037] Similarly, the second cold weights include weight combinations with a co-occurrence rate greater than or equal to the second threshold, and these weight combinations are stored in the same storage unit in the second accelerator.

[0038] In this application, the first accelerator stores the weight combination in the same storage unit, thereby enabling the first accelerator to retrieve the weight combination from a smaller storage area, which speeds up the retrieval of weights and improves the model inference efficiency.

[0039] In one possible scenario, the aforementioned weight combination may include at least two weights from the weight matrix corresponding to the model.

[0040] In one possible scenario, the aforementioned storage unit is a data block, cache line, data segment, or physical storage block.

[0041] In one possible implementation, the above model inference method further includes: a first accelerator acquiring a first weight switching request to indicate a switch of the first cold weights configured in the first accelerator, and then switching the third sub-cold weight among the first cold weights to a hot weight according to the first weight switching request. The activation frequency of the third sub-cold weight is greater than the activation frequency of the other weights among the cold weights excluding the third sub-cold weight.

[0042] In this application, the first accelerator reduces the amount of data that the first accelerator needs to obtain weights from the more distant first cold weights when the storage area of ​​the first cold weights is farther than the storage area of ​​the first hot weights. This reduces the time spent by the first accelerator in obtaining weights and thus improves the efficiency of model inference.

[0043] In one possible implementation, the above model inference method further includes: a first accelerator acquiring a second weight switching request to indicate a switch of the first hot weights configured in the first accelerator, and then switching a third sub-hot weight in the first hot weights to a cold weight according to the second weight switching request. The activation frequency of the third sub-hot weight is lower than the activation frequencies of the other weights in the first hot weights besides the third sub-hot weight.

[0044] In this application, the first accelerator reduces the amount of data stored in the first hot weight by switching the third sub-hot weight in the first hot weight to a cold weight, thereby reducing the resource load and power consumption of the first accelerator.

[0045] In one possible implementation, the above model inference method further includes: a first accelerator acquiring a third weight switching request indicating a switch of weights configured in the first accelerator; and then, based on the third weight switching request, switching a third sub-hot weight in the first hot weights to a cold weight, and switching a third sub-cold weight in the first cold weights configured in the first accelerator to a hot weight. The activation frequency of the third sub-hot weight is lower than the activation frequency of other weights in the first hot weights besides the third sub-hot weight, and the activation frequency of the third sub-cold weight is higher than the activation frequency of other weights in the cold weights besides the third sub-cold weight.

[0046] In this application, the first accelerator reduces the amount of data the first accelerator needs to acquire weights from the more distant first cold weights (e.g., reducing the amount of data acquired from the third sub-cold weights) by switching the third sub-cold weight in the first hot weights to a cold weight and switching the third sub-cold weight in the first cold weights to a hot weight. This reduces the time the first accelerator spends acquiring weights and improves the efficiency of model inference.

[0047] In one possible implementation, the above-described model inference system further includes a processor for managing the plurality of accelerators. The process of the model inference system acquiring a first inference request includes: the processor acquiring the first inference request, then performing flyline prediction on the first inference request, determining the weights required by the first and second accelerators to process the first inference request, and then sending an identifier of the weights required by the first accelerator to process the first inference request to the first accelerator, and sending an identifier of the weights required by the second accelerator to process the first inference request to the second accelerator.

[0048] In this application, the processor sends the identifiers of the weights required by the first accelerator and the second accelerator to the first accelerator when processing the first inference request. This enables the first accelerator and the second accelerator to selectively activate some weights in the model, thereby reducing the number of weights acquired by the first accelerator and the second accelerator when processing the first inference request, reducing the time required to acquire weights, and thus improving the efficiency of the model inference system in performing model inference.

[0049] In one possible scenario, the aforementioned identifier could be the storage address of the weights to be utilized.

[0050] In one possible implementation, the processor is further configured with a counting matrix, which indicates the activation frequency of weights in the model. The model inference method further includes: the processor determining the first position of the required weight in the weight matrix based on the weights needed by the model inference system to process the first inference request, and then incrementing the value at the second position corresponding to the first position in the counting matrix by 1. If the value at the third position in the counting matrix is ​​greater than a third threshold, a first weight switching request is sent to the first accelerator; if the value at the fourth position in the counting matrix is ​​less than a fourth threshold, a second weight switching request is sent to the first accelerator; if the value at the third position in the counting matrix is ​​greater than the third threshold and the value at the fourth position in the counting matrix is ​​less than the fourth threshold, a third weight switching request is sent to the first accelerator.

[0051] In this application, during the processing of inference requests, the processor counts the number of activations of the required weights. Then, when the threshold requirements (third threshold, fourth threshold) are met, it sends a weight switching request (first weight switching request, second weight switching request, third weight switching request) to the first accelerator. The processor dynamically switches between cold and hot weights according to the actual inference request, so that the first accelerator stores the latest first hot weight. This reduces the amount of weight data that the first accelerator needs to obtain from other devices in the model inference system when processing inference requests, thereby reducing the time spent by the first accelerator in obtaining weights and improving the inference efficiency of the model.

[0052] In one possible scenario, the weights in the weight matrix corresponding to the third position are configured in the first accelerator, and the weights in the weight matrix corresponding to the fourth position are configured in the first accelerator.

[0053] Secondly, this application provides a model inference system. The model inference system includes various devices for executing the model inference method in the first aspect or any optional implementation thereof. In one possible example, the model inference system includes a plurality of accelerators, including a first accelerator and a second accelerator, wherein a first hot weight of the model is configured in the first accelerator and a second hot weight of the model is configured in the second accelerator.

[0054] The model inference system is used to obtain the first inference request.

[0055] A first accelerator is used to process the first inference request according to the first hot weights to obtain a first inference result; the activation frequency of the first hot weights is greater than the activation frequency of the cold weights in the model.

[0056] A second accelerator is used to process the first inference request according to the second heat weight to obtain a second inference result; the second heat weight is the same as the first heat weight.

[0057] The first accelerator is used to obtain the output result based on the first inference result and the second inference result.

[0058] In one possible implementation, the first accelerator and the second accelerator are of the same type.

[0059] In one possible implementation, the aforementioned accelerators further include a third accelerator, which is of a different type from the first and second accelerators. The third accelerator is configured with third hot weights for the model, which are different from the first hot weights. The third accelerator is used to process the first inference request according to the third hot weights to obtain a third inference result. The first accelerator is specifically used to obtain the output result based on the first inference result, the second inference result, and the third inference result.

[0060] In one possible implementation, the first accelerator is further configured with a first cold weight from the cold weights, and the second accelerator is further configured with a second cold weight from the cold weights. Specifically, the first accelerator processes the first inference request based on the first hot weight and the first cold weight to obtain a first inference result. Specifically, the second accelerator processes the first inference request based on the second hot weight and the second cold weight to obtain a second inference result.

[0061] In one possible implementation, the first accelerator is further configured to process the first inference request based on the first sub-hot weight in the first hot weight and the first sub-cold weight in the first cold weight to obtain a first inference result. The second accelerator is further configured to process the first inference request based on the second sub-hot weight in the second hot weight and the second sub-cold weight in the second cold weight to obtain a second inference result.

[0062] In one possible implementation, the difference between the sum of the data amounts of the first sub-hot weight and the first sub-cold weight and the sum of the data amounts of the second sub-hot weight and the second sub-cold weight is less than or equal to a first threshold.

[0063] In one possible implementation, the first cold weights include weight combinations with a co-occurrence rate greater than or equal to a second threshold, and the weight combinations are stored in the same storage unit in the first accelerator.

[0064] In one possible implementation, the first accelerator is further configured to acquire a first weight switching request; the first weight switching request is used to instruct a switching of a first cold weight configured in the first accelerator. The first accelerator is further configured to switch a third sub-cold weight among the first cold weights to a hot weight according to the first weight switching request; the activation frequency of the third sub-cold weight is greater than the activation frequency of other weights among the cold weights excluding the third sub-cold weight.

[0065] In one possible implementation, the first accelerator is further configured to acquire a second weight switching request; the second weight switching request is used to instruct the switching of a first hot weight configured in the first accelerator. The first accelerator is further configured to switch a third sub-hot weight in the first hot weight to a cold weight according to the second weight switching request; the activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weight except for the third sub-hot weight.

[0066] In one possible implementation, the first accelerator is further configured to acquire a third weight switching request; the third weight switching request is used to instruct a switching of weights configured in the first accelerator. The first accelerator is further configured to, according to the third weight switching request, switch a third sub-hot weight in the first hot weights to a cold weight, and switch a third sub-cold weight in the first cold weights configured in the first accelerator to a hot weight. The activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weights except for the third sub-hot weight, and the activation frequency of the third sub-cold weight is greater than the activation frequency of other weights in the cold weights except for the third sub-cold weight.

[0067] For more detailed implementation information about the model inference system, please refer to the description of the first aspect or any of the implementation methods in the first aspect above, as well as the content of the specific implementation methods below, which will not be repeated here.

[0068] Thirdly, this application provides a chip, including: a control circuit and an interface circuit. The interface circuit is used to acquire a first inference request, and the control circuit is used to execute the processing of the first request by the first accelerator or the second accelerator in the method of the first aspect or any possible implementation of the first aspect. The chip provided in this aspect may be a chip included in the first accelerator or the second accelerator in the first aspect for performing computation during model inference.

[0069] Fourthly, this application provides a server. The server includes a processor, multiple accelerators, and memory.

[0070] The processor is used to execute instructions stored in memory to retrieve the first inference request.

[0071] The first accelerator among multiple accelerators is configured with the first hot weights of the model. The first accelerator processes the first inference request based on the first hot weights and obtains the first inference result. The activation frequency of the first hot weights is greater than the activation frequency of the cold weights in the model.

[0072] The second accelerator among the multiple accelerators is configured with the second hot weights of the model. The second accelerator processes the first inference request according to the second hot weights to obtain the second inference result; the second hot weights are the same as the first hot weights.

[0073] The first accelerator obtains the output result based on the first and second inference results.

[0074] Fifthly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, each computing device including a memory and a processor. The memory stores computer instructions; when the processor executes the computer instructions, it implements the method described in the first aspect or any possible implementation of the first aspect. The computing device cluster provided in this aspect can be the model inference system described in the second aspect above, that is, the computing device cluster includes a first accelerator and a second accelerator.

[0075] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processing device, implement the method described in the first aspect or any possible implementation of the first aspect.

[0076] In a seventh aspect, this application provides a computer program product comprising a computer program or instructions that, when executed by a processing device, implement the method in the first aspect or any possible implementation thereof.

[0077] The beneficial effects of aspects two through seven above can be referred to in aspect one or any possible implementation of aspect one, and will not be elaborated here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. Attached Figure Description

[0078] Figure 1 A schematic diagram of the structure of a data processing system provided in this application; Figure 2 A schematic diagram of a distributed reasoning model provided in this application; Figure 3 A flowchart illustrating a model reasoning method provided in this application. Figure 1 ; Figure 4 A schematic diagram for determining hot and cold weights provided in this application; Figure 5 A schematic diagram illustrating the training of an activation classifier provided in this application; Figure 6 A schematic diagram of a hot and cold weight configuration provided in this application Figure 1 ; Figure 7 A schematic diagram of a hot and cold weight configuration provided in this application Figure 2 ; Figure 8 A schematic diagram of load balancing distribution provided in this application; Figure 9 A flowchart illustrating a hot / cold weight switching method provided in this application; Figure 10 A flowchart illustrating a model reasoning method provided in this application. Figure 2 ; Figure 11 This is a schematic diagram of an accelerator provided in this application. Detailed Implementation

[0079] First, let's introduce the technical terms used in this application.

[0080] Large language models (LLMs) leverage their powerful computing capabilities and sophisticated algorithms to effectively process massive amounts of data, providing users with efficient and accurate information processing and analysis services. LLMs not only excel in understanding and generating human language but also demonstrate immense potential in solving complex problems and tasks. For example, LLMs are widely used in automated question-answering systems, text summarization, machine translation, and language generation, significantly improving efficiency and accuracy. Particularly when dealing with large-scale datasets, LLMs can uncover deep patterns and associations, supporting decision-making. Furthermore, the self-learning ability of LLMs allows them to continuously evolve, constantly improving their performance and intelligence by learning from new data. Commonly used LLM models include the Transformer-based Bidirectional Encoder Representation (BERT) model, which pre-trains a deep bidirectional representation using unlabeled text by jointly adjusting left and right contexts across all layers. In some optional cases, LLMs may also be simply referred to as large models. The large model provided in this application can refer not only to LLMs, but also to models with a certain number of parameters. Depending on the domain in which the large model is applied, it can also refer to a model that includes various functions such as image processing, human-computer interaction, semantic search, semantic query, or dialogue. This application does not limit the domain in which the large model can be applied or its specific name. In this document, for the sake of simplicity, it is referred to as a large model, but this should not be construed as a limitation of this application, and will not be elaborated further thereafter.

[0081] Tensor parallelism (TP) refers to splitting the weights of each network layer into multiple parts and allocating one part of each weight to an accelerator. By deploying the model across multiple accelerators using tensor parallelism, each accelerator processes the input data separately, producing multiple processing results. These multiple processing results are then aggregated to obtain the output corresponding to the input data.

[0082] Data parallelism (DP) refers to dividing data into multiple batches or parts, and then using multiple accelerators deployed with the same model to process each batch or part of the data separately, with one accelerator processing one batch or part of the data. Alternatively, it can use multiple accelerators deployed with the same network layer in the model to process multiple batches or parts of the data separately, with one accelerator processing one batch or part of the data.

[0083] Pipeline parallelism (PP) refers to configuring each network layer in the model onto a different accelerator. After the input data is fed into the accelerator that deploys the first network layer of the model, the processing result of the input data by the aforementioned accelerator is fed into the accelerator that deploys the second network layer of the model, and so on, until the output result is obtained after processing by the accelerator that deploys the last network layer of the model.

[0084] Currently, LLMs based on the transformer structure have reached trillions of parameters (also known as weights). During the process of using LLMs to perform model inference, based on the input data input by the tenant, such as the "query statement", LLMs can generate corresponding output results, such as the answer to the "query statement", through autoregression.

[0085] Because LLMs have large weights and are limited by the memory wall of a single GPU (accelerator), they are often deployed across multiple GPUs using methods such as TP, DP, and PP, such as multi-machine, multi-GPU inference servers or supernodes. Since the weight matrix in an LLM is typically a slender matrix, accelerators using slender matrices for model inference exhibit low computational cost but high memory access requirements, leading to longer times when the accelerator retrieves the weight matrix and impacting model inference efficiency.

[0086] To address the above issues, two possible solutions are presented below.

[0087] Solution 1 reduces model weights by quantizing and / or pruning the LLM, thereby reducing the amount of data the accelerator needs to acquire the weight matrix, which in turn reduces the time spent acquiring the weight matrix and improves model inference efficiency. However, LLM differs significantly from traditional models (such as convolutional neural networks and recurrent neural networks) in network structure, and directly reusing traditional quantization / pruning methods results in a significant loss of inference accuracy for LLM.

[0088] Solution 2 involves deploying a model across more accelerators using methods such as TP, DP, and PP to reduce the model weights configured in a single accelerator. However, more accelerators lead to more discrete model weights, resulting in higher communication costs (increased data volume for communication between accelerators). This causes accelerators to acquire weights more slowly, and consequently, the efficiency of model inference across multiple accelerator cards does not change significantly.

[0089] Based on this, this application provides a model inference method. This model inference method is applied to a model inference system, including a first accelerator and a second accelerator. The first accelerator is configured with first hot weights of the model, and the second accelerator is configured with second hot weights of the model. The model inference system obtains a first inference request. The first accelerator processes the first inference request according to the first hot weights to obtain a first inference result, and the second accelerator processes the first inference request according to the second hot weights to obtain a second inference result. Then, the first accelerator obtains an output result based on the first and second inference results. The first hot weights and second hot weights are the same, the activation frequency of the first hot weights is greater than the activation frequency of the cold weights in the model, and the first hot weights and second hot weights are identical.

[0090] In this application, both the first and second accelerators are equipped with the same hot weights. Therefore, when performing model inference, the first and second accelerators can obtain the hot weights locally for model inference. Since the accelerators (first and second) obtain hot weights locally at a faster rate than they obtain weights from other locations (such as the processor), the efficiency of model inference can be improved. Furthermore, since the activation frequency of hot weights is greater than the activation frequency of cold weights in the model, when the first and second accelerators process the first inference request, they can obtain most of the weights required to execute the first inference request locally. This reduces the amount of data the first and second accelerators need to obtain weights from other locations. Furthermore, since the accelerators obtain hot weights locally at a faster rate than they obtain weights from other locations, the efficiency of model inference can be further improved.

[0091] Next, the model reasoning method provided in this application will be described in detail with reference to the accompanying drawings.

[0092] Figure 1 Figure 1 shows a schematic diagram of a data processing system provided in this application. The data processing system includes a computing device 110, a model training system 120, a database 130, a model inference system 140, a terminal device 150, a data storage system 160, and a data acquisition device 170.

[0093] The computing device 110 can be used to interact with the user, such as by interacting with the user-side terminal device 150 to obtain input data from the user. In one possible scenario, the computing device 110 can also be used to preprocess the input (such as data cleaning, noise reduction, etc.) to obtain preprocessed data, and then send the preprocessed data to the model inference system 140.

[0094] In one possible scenario, the computing device 110 can also perform embedded representation on the preprocessed data to obtain the corresponding vector.

[0095] When the input data is a query statement, the computing device 110 can perform data cleaning, noise reduction, etc. on the query statement to obtain a preprocessed query statement, and then perform embedded expression on the preprocessed query statement to obtain the corresponding vector.

[0096] In another possible scenario, computing device 110 is also used to preprocess the training data sent by terminal device 150, and then store the preprocessed training data in database 130.

[0097] In one possible scenario, the data storage system 160 can be used to temporarily store the preprocessed data, vectors, or output results from the model inference system 140. Furthermore, the computing device 110 can use the input data and corresponding output results as input data again and store them in the database 130.

[0098] Terminal device 150 is used to receive user operations, such as receiving user input data or training data.

[0099] The training data stored in database 130 comes not only from the preprocessed training data of computing device 110, but may also include training data collected by data acquisition device 170. Data acquisition device 170 is used to collect training data and store the training data in database 130.

[0100] The model training system 120 is used to train the neural network 101. For example, the model training system 120 uses training data in the database 130 to train the neural network until the loss function in the neural network converges and the loss function value is less than a certain threshold, at which point the neural network training is complete, thereby enabling the neural network 101 to reach a certain accuracy. In one possible example, the model training system 120 can deploy the trained neural network 101 into the model inference system 140. The neural network 101 can be the large model described above.

[0101] The model inference system 140 deploys a neural network 101, which then performs model inference based on the input data provided by the user to obtain the output result. In other words, the model inference system 140 inputs the input data into the neural network 101 and then outputs the result.

[0102] The model inference system 140 can return the above output results to the terminal device 150, thereby providing them to the user so that the user can view the output results.

[0103] Taking the input data as a query statement as an example, the model reasoning system 140 inputs the query statement into the neural network 101 to obtain the query result corresponding to the query statement, and then returns the query result to the terminal device 150.

[0104] In one possible scenario, the model inference system 140 may receive input data sent by the terminal device 150.

[0105] In another possible scenario, the model inference system 140 may receive preprocessed input data sent by the computing device 110.

[0106] It should be noted that in practical applications, the training data maintained in database 130 may not all come from data acquisition device 170; it may also be received from other devices. Furthermore, the model training system 120 may not necessarily train the neural network 101 entirely based on the training data maintained in database 130; it may also obtain training data from the cloud or other sources to train the neural network 101. The above description should not be construed as limiting the embodiments of this application.

[0107] In this embodiment, the neural network 101 described above can be referred to as an AI model or model, which may refer to an LLM or other models, etc.

[0108] In one alternative example, neural network 101 refers to an LLM. Since LLMs contain many processing layers, training / inference on a single device is time-consuming. Therefore, a distributed training approach is used for training / inference on the LLM. See below for details. Figures 2 to 10 The provided embodiments are not described in detail here.

[0109] In another alternative example, neural network 101 refers to other types of networks. For example, neural network 101 is a convolutional neural network (CNN), a recurrent neural network (RNN), or a graph neural network (GNN), etc. Further implementations of CNNs, RNNs, or GNNs can be found in the description of commonly used techniques, which will not be elaborated upon here.

[0110] The functions of computing device 110, model training system 120, database 130, model inference system 140, terminal device 150, data storage system 160, and data acquisition device 170 have been described above. The possible structures of computing device 110, model training system 120, database 130, model inference system 140, terminal device 150, data storage system 160, and data acquisition device 170 will be described below.

[0111] The computing device 110 can be a terminal, such as a computer, mobile terminal, tablet computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (ER) device, camera, or in-vehicle computer, or it can be an edge device (e.g., a box carrying a chip with processing capabilities). In this application, the computing device 110 can be a computing device connected to a base station, or a computing device deployed at a base station, such as a server or cloud device.

[0112] The model training system 120 may include one or more processing devices, which may be terminals or other computing devices that support integer or floating-point calculations, such as servers, cloud devices or chip systems.

[0113] In one possible embodiment, the computing device 110 and the model training system 120 include different physical devices for processing. Exemplarily, the computing device 110 and the model training system 120 include chip systems on different physical devices. For example, the computing device 110 can be a GPU, a central processing unit, other general-purpose processors, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0114] In another possible embodiment, the computing device 110 and the processing device included in the model training system 120 are deployed on the same physical device, or the computing device 110 and the processing device included in the model training system 120 are on the same physical device.

[0115] The data acquisition device 170 and the processing devices included in the computing device 110 and model training system 120 may be the same or different devices. In the embodiments of this application, the data acquisition device 170 may be a camera, mobile phone, tablet computer, or a computing device or network device with data acquisition function.

[0116] In some embodiments, if the computing device 110 and the processing device included in the model training system 120 are the same physical device, the model training system 120 can configure the trained neural network 101 to itself and use the trained neural network 101 to achieve the target function to be achieved by the model, such as the function of signage or document verification in commercial areas, schools, parks, and sports venues in cities, as well as the operation of target detection, object recognition or classification on data, and the function of facial recognition payment and object classification (such as commodity classification).

[0117] In other embodiments, the model training system 120 can configure the trained neural network 101 into the model inference system 140, which includes multiple accelerators, to enable the model inference system to achieve the target functions to be performed by the model described above. In addition to the target functions described in the above embodiments, the neural network 101 can also achieve some functions achievable through LLM, etc.

[0118] Furthermore, based on the functions performed by the computing device 110, the computing device 110 can be further subdivided into, for example... Figure 1 The architecture shown is as follows: Figure 1 As shown, the computing device 110 is equipped with an I / O interface 111 and a preprocessing module 112.

[0119] I / O interface 111 is used for data interaction with external devices. Users can input data into I / O interface 111 via terminal device 150, for example, to instruct computing device 110 to initiate feature extraction or model training on training data. Alternatively, the input data can also come from database 130.

[0120] The preprocessing module 112 is used to preprocess the input data received from the I / O interface 111. In this embodiment, the preprocessing module 112 can be used to generate training data, such as a training set, a validation set, and a test set, based on the input data received from the I / O interface 111. Optionally, the preprocessing module 112 can also perform preprocessing operations such as denoising to remove irrelevant information and recover useful real information from the input data, such as the feature vector corresponding to the query statement.

[0121] During the preprocessing and other related processes of the input data by the computing device 110, the computing device 110 can call the data, code, etc. in the data storage system 160 for corresponding processing, and can also store the data and instructions obtained from the corresponding processing into the data storage system 160.

[0122] Finally, the I / O interface 111 sends the processing result to the model inference system 140, so that the model deployed in the model inference system 140 can be used to further process the aforementioned processing result to obtain the output result.

[0123] Terminal device 150 can serve as a data acquisition end, collecting data such as... Figure 1 The input data and output results of the input I / O interface 111 are used as new sample data and stored in the database 130. Alternatively, the sample data can be collected without going through the terminal device 150, but rather directly by the I / O interface 111. Figure 1 The input data of the input I / O interface 111 and the output results of the output I / O interface 111 are stored in the database 130 as new sample data.

[0124] The model inference system 140 may include processing devices such as GPU, NPU, TPU, central processing unit (CPU), other general-purpose processors, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0125] In one possible scenario, the model inference system 140 includes a processor 140a and multiple accelerators, such as accelerator a 141, accelerator b 142, accelerator c 143 and accelerator d 144.

[0126] exist Figure 1 In the illustrated model inference system 140, processor 140a is used to divide the model into multiple modules, each module including a portion of each network layer in the model, and then deploy these multiple modules to multiple accelerators, with one module deployed in each accelerator, thereby improving the model's inference efficiency. Processor 140a can be a CPU used to receive query statements and send the feature vectors corresponding to the query statements to different accelerators, etc. In terms of hardware implementation, processor 140a can include one or more processing chips, which is not limited in this application.

[0127] The accelerators provided in this application include one or more dies, such as accelerator a 141 including die 1 and die 2, accelerator b 142 including die 3 and die 4, accelerator c 143 including die 5 and die 6, and accelerator d 144 including die 7 and die 8. Taking accelerator a 141 in accelerators a 141, b 142, c 143, and d 144 as an example, accelerator a 141 can be, but is not limited to, a GPU chip, an NPU chip, a TPU chip, a microprocessor chip, an ASIC chip, or one or more integrated circuit chips used to control the execution of the computer program product provided in this application. Each accelerator includes a die that is a GPU within a GPU chip, an NPU within an NPU chip, or a TPU within a TPU chip. For example, if the die is a GPU within a GPU chip, then the die can be used to perform mathematical and geometric calculations to achieve tasks such as image rendering.

[0128] In some possible implementations, different accelerators are networked in a full mesh mode, meaning that each accelerator is directly connected to the others. In some cases, the direct connection between different accelerators can be a chip-to-chip communication link, such as using one or more of the following methods: high-speed custom communication system (Huawei). ® Cache coherence system (HCCS) interface, high-speed interconnect bandwidth interface, integrated circuit bus, controller area network bus, serial peripheral device interface, queued serial peripheral device interface, full-duplex asynchronous serial interface or half-duplex differential serial interface, etc.

[0129] The HCCS interface is a high-speed connection channel between dies, used to accelerate data and computation to produce executable results. For example, in an accelerator, different dies are connected in pairs using HCCS technology.

[0130] High-speed interconnect bandwidth is a high-speed interconnect technology between GPUs, which is usually achieved through multiple pairs of wires printed on a computer board, with each pair of wires connecting to different GPUs.

[0131] The Inter-Integrated Circuit (I2C) bus is a source-synchronous serial bus used for short-distance communication between different integrated circuits. I2C uses two lines for data transmission: a serial data line (SDL) and a serial clock line (SCL). The Controller Area Network (CAN) bus is a serial communication protocol bus used for real-time applications; it can use twisted-pair cables to transmit signals. The Serial Peripheral Interface (SPI) bus is a 3-wire synchronous serial full-duplex communication interface with advantages such as simple circuitry, high speed, and reliable communication. The Queued Serial Peripheral Interface (QSPI) bus adds a queued transmission mechanism to SPI; QSPI uses a dedicated communication interface to connect single, dual, or four data lines. A full-duplex asynchronous serial interface, also known as a universal asynchronous receiver / transmitter (UART) interface, is a universal serial data bus used for asynchronous communication. This UART bus can be a bidirectional communication bus, converting the data to be transmitted between serial and parallel communication modes; for example, a UART interface refers to an RS-232 interface. A half-duplex differential serial interface is a serial communication bus interface that uses a two-wire, differential transmission, and half-duplex mode, such as an RS-485 interface.

[0132] It is worth noting that, unlike chip-to-chip communication links, in an accelerator, the communication links between individual chips can be connected using, but are not limited to, SIO interfaces.

[0133] In this embodiment, the transmission bandwidth (SIO bandwidth) of the die-to-die communication link is greater than the chip-to-chip transmission bandwidth. Transmission bandwidth refers to the maximum amount of data that different devices can transmit per unit time. For example, different dies within the same accelerator are connected via an SIO interface, and the transmission bandwidth (SIO bandwidth) of the communication link (or channel) connected via SIO is greater than the transmission bandwidth of the communication link between different chips.

[0134] Taking accelerators a 141 and b 142 as shown in Figure 1 as an example, the SIO bandwidth between chip 1 and chip 2 is greater than the transmission bandwidth between accelerators a 141 and b 142. For example, the SIO bandwidth between chip 1 and chip 2 is 392 gigabytes per second (GB / s), while the transmission bandwidth between accelerators a 141 and b 142 is 100 GB / s.

[0135] Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. Figure 1 The positional relationships between the devices, components, modules, etc., shown are not limiting. Depending on the user's needs for model inference, the model inference system 140 and computing device 110 may include more or fewer hardware components; this application does not limit this. For example, in Figure 1 In this context, the data storage system 160 is an external storage device relative to the computing device 110. In other cases, the data storage system 160 may also be placed within the computing device 110.

[0136] Below Figure 1 Based on the model inference system 140 shown, assuming that the model inference system 140 of this application embodiment is deployed with an AI model based on the transformer architecture, the following will be combined with... Figure 2 One possible example is shown. Figure 2 This is a schematic diagram of a distributed inference model provided in this application. Please refer to [link / reference]. Figure 2 The model includes multiple transformer modules, such as transformer module 1, transformer module 2, and transformer module n, where n is a positive integer. These multiple transformer modules are networked in a fully interconnected mode (full mesh). The complete transformer is broken down into multiple transformer modules, and each transformer module includes a part of each network layer in the complete transformer.

[0137] Different transformer modules depend on Figure 1The model inference system 140 shown implements this, where each transformer module is configured into a different accelerator. Multiple accelerators work together to process the input data, thereby improving the model's inference efficiency. For example, different transformer modules in the model are configured into different accelerators for inference. For instance, processor 140a configures transformer module 1 into accelerator a 141, transformer module 2 into accelerator b 142, and transformer module n into accelerator d 144. Other transformer modules can also be configured into other accelerators in the model inference system 140, which will not be elaborated here.

[0138] During the model's inference process, processor 140a acquires a sequence and inputs it into the model. Different accelerators then use the transformer module to perform inference based on this sequence and output the corresponding inference result. It can be understood that different transformer modules in the model are inferred by different accelerators, thus achieving parallel inference of the model.

[0139] For example, the sequence is the vector corresponding to the query statement. Then, the processor 140a sends the vector to different accelerators for processing, so that each accelerator can use the transformer module to reason about the vector and obtain the corresponding reasoning results. Then, the reasoning results of each accelerator are summarized to obtain the output result, such as the query result.

[0140] A sequence is an ordered collection of data, where each element has a unique position. In some optional examples, sequences are mutable, allowing some or all elements to be modified; and sequences can be traversed iteratively, with devices able to slice sub-sequences. Sequences can be used to perform one or more operations, such as adding, deleting, sorting, and searching. For example, elements in a sequence can be of any type, such as numbers, strings, objects, arrays, tuples, linked lists (or lists), tokens, stacks, and queues. A string is a sequence of characters, commonly used in text processing, cryptography, and image processing. An array is a sequence of elements of the same type, often used for storing large amounts of data and performing numerical calculations. A linked list is a sequence of nodes, often used to implement dynamic data structures and efficient insertion and deletion operations. A tuple is an immutable sequence, often used to group multiple values ​​into a single unit. A list is a mutable sequence, often used for storing and manipulating data. Stacks and queues are special types of sequences, often used to implement data structures and algorithms.

[0141] Please see Figure 2 Taking a sequence as a vector for a query statement as an example, the query statement is semantic information generated by processor 140a based on a sequence of data such as text or audio information, which may be input by the user or transmitted by other devices. The query statement includes multiple words (tokens), such as token1 and token2 corresponding to the black pattern. In this document, tokens may include, but are not limited to: words, characters, punctuation marks, special symbols (such as calculation symbols), etc. In some optional cases, words may also be referred to as tokens, which is not limited in this application.

[0142] For example, the input data is text or audio information, etc. The processor 140a extracts semantic features from the input data to obtain the semantic information of the input data, i.e., the query statement. Then the processor 140a performs embedded expression on the query statement to obtain a sequence.

[0143] In other embodiments of this application, the above sequence may also be obtained by processing the input data by the computing device 110.

[0144] As users' demands for model processing capabilities gradually increase, the number of words in query statements continues to grow. When deploying the model on a single device to process long sequences, the device's processing efficiency is low. In this example, a long sequence refers to a query statement containing a large number of words.

[0145] The following uses accelerator a 141 as an example to briefly explain the inference process of the model. Accelerator a 141 calculates the query vector stored locally in accelerator a 141 and all key-value vectors associated with the query vector in the model inference system 140 to obtain the inference result 1.

[0146] Regarding the possible implementations of the query vector (Q), key vector (K), and value vector (V), the following explanation uses a word in the query statement (such as token1) as an example: The processor 140a performs vector transformation on token1 to obtain the Q, K, and V corresponding to token1. For example, the process of performing vector transformation on token1 satisfies the following formulas (1) to (3).

[0147] Formula (1) Formula (2) Formula (3) Where X is the embedding representation of token 1, that is, the abstract representation of token 1 mapped to a low-dimensional space; Q is the query vector, This is the weight matrix (weights) corresponding to the query vector, where K is the key vector. V represents the weights corresponding to the key vectors, and V represents the value vectors. These are the weights corresponding to the value vectors.

[0148] In this paper, for the sake of simplicity, key vectors and value vectors can be simply referred to as key-value vectors.

[0149] Please continue reading. Figure 2 Different accelerators are used for different transformer modules in the inference model, and the model parameters of different transformer modules are different, such as the weights stored in different accelerators. Since the input data of the model is a query statement, and the model is split into multiple transformer modules, each transformer module includes a part of each network layer in the complete transformer, the processor 140a inputs the query statement into each transformer module for inference, thereby reducing the amount of computation that a single accelerator needs to perform, improving the computational efficiency of these accelerators on the input data, and thus improving the model inference efficiency.

[0150] For example, transformer module 1 includes an attention layer 11 and a multilayer perceptron (MLP) 11. Attention layer 11 is part of the complete attention layer in the model. It is used to calculate the query vector, key vector, and value vector corresponding to the input data, and finally outputs the result of attention layer 11. MLP 11 is part of the complete MLP in the model. MLP 11 includes two linear layers used to linearly process the output of attention layer 11 to obtain the inference result of transformer module 1 (inference result 1). In other words, transformer module 1 includes the weights corresponding to the complete attention layer and the weights corresponding to the complete MLP. Figure 2 The contents and inference methods of other transformer modules shown can be found in the description of transformer module 1. Regarding the specific network structure of the transformer modules deployed in the accelerator, the following section will discuss... Figure 2 An example is provided.

[0151] Please see Figure 2The transformer module consists of two parts: an encoder and a decoder.

[0152] An encoder processes input data, transforming it into an internal representation (such as an encoding matrix) that the model can understand. An encoder consists of multiple identical layers stacked on top of each other, each layer containing two main substructures: a multi-head self-attention mechanism and a feed-forward neural network. For example... Figure 2 As shown, the encoder includes: a multi-head self-attention layer 11, a residual connection and add-layer normalization (Add&Norm) operation layer (11), a feedforward layer (11), and Add&Norm (21). The multi-head self-attention layer 11 is a network processing layer determined based on the multi-head self-attention mechanism. This multi-head self-attention layer 11 is used to: calculate the dot product between the Q corresponding to a word (token) and the other K and V associated with that Q, obtaining the relevance score between each word and other words. The feedforward layer (11) is a network processing layer determined based on a feedforward neural network. This feedforward layer (11) is used to feed forward the model parameters. Add&Norm (11) and Add&Norm (21) are used to prevent network layer degradation in the encoder and to normalize the activation values ​​of different network layers, such as by weighted summation of the relevance scores determined by the multi-head self-attention layer 11, obtaining a new representation for each word, such as the encoding matrix.

[0153] The decoder generates inference results (such as inference result 1) based on the encoding matrix generated by the encoder. The encoder includes: a mask multi-head self-attention layer 11, Add&Norm (31), a multi-head self-attention layer 21, Add&Norm (41), a feedforward layer (21), Add&Norm (51), a linear processing layer 11, and a normalization processing layer 11. The functions of Add&Norm (31) to Add&Norm (51) can be referred to the description of the encoder mentioned above, and will not be repeated here. The mask multi-head self-attention layer 11 differs from the multi-head self-attention layer 21 in that: the mask multi-head self-attention layer uses a masked operation, the KV of the multi-head self-attention layer 21 is calculated using the encoding matrix of the encoder, and the Q is calculated using the output of the previous encoding block (decoderblock). The previous encoding block of the multi-head self-attention layer 21 refers to the decoder corresponding to another word before predicting token1. The linear processing layer and the normalization processing layer are used to perform linear processing and normalization operations, and then output the prediction result corresponding to token1, such as inference result 1.

[0154] It is worth noting that the encoder shown above is a partial network structure of a complete encoder, and the decoder is a partial network structure of a complete decoder. For example, the multi-head self-attention layer 11 is a part of the multi-head self-attention layer included in the complete encoder, and the accelerator a 141 stores a partial weight in the weight matrix of the complete multi-head self-attention layer, that is, a partial weight corresponding to the multi-head self-attention layer 11.

[0155] Figure 2 The structure of the transformer model shown enables the model to handle sequence-to-sequence (seq2seq) tasks, such as machine translation.

[0156] The following illustrates a possible embodiment of the model inference method, which can be applied to... Figure 1 The data processing system shown is, for example, applied in model inference system 140 within a data processing system. For example... Figure 3 As shown, Figure 3 A flowchart illustrating a model reasoning method provided in this application. Figure 1 The following explanation uses the execution of the model inference method by devices in a model inference system as an example. The first accelerator can be referred to as accelerator a 141, which contains the model's first thermal weights. The second accelerator can be referred to as accelerator b 142, which contains the model's second thermal weights. Figure 3As shown, the model inference method may include the following steps S310 to S340.

[0157] S310, Model Inference System 140 obtains the first inference request.

[0158] In one possible scenario, the first inference request carries the aforementioned input data, such as a sequence.

[0159] In one possible example, the above sequence could be a query statement.

[0160] Regarding the acquisition of the content of the first inference request by the model inference system 140, two possible implementation methods are provided below.

[0161] In one possible implementation, the model inference system 140 acquires a first inference request, including: processor 140a acquiring the first inference request.

[0162] In one possible example, processor 140a obtains the first inference request via transceiver.

[0163] In one possible scenario, processor 140a will distribute the first inference request it receives to accelerator a141 and accelerator b142.

[0164] In one possible scenario, processor 140a performs flyline prediction on the first inference request to determine the weights required when multiple accelerators in the model inference system process the first inference request. These required weights may include a first sub-hot weight within a first hot weight and a second sub-hot weight within a second hot weight.

[0165] Optionally, the weights required for the aforementioned accelerators may also include a first sub-cold weight in the first cold weight and a second sub-cold weight in the second cold weight. The first cold weight is configured in accelerator a 141, and the second cold weight is configured in accelerator b 142.

[0166] Then, processor 140 sends the identifier of the corresponding weight to each of the multiple accelerators according to the weight to be utilized. The aforementioned identifier may be the storage address of the weight, the position of the weight in the weight matrix, etc. The position of the weight in the weight matrix can be directly predicted by flying lines.

[0167] For details regarding this situation, please refer to the following: Figure 6 The specific wording is not elaborated here.

[0168] In another possible implementation, the model inference system 140 acquires a first inference request, including accelerator a 141 and / or accelerator b 142 acquiring the first inference request.

[0169] In one possible example, accelerator a 141 is the main accelerator in the model inference system 140, which is responsible for coordinating and integrating the outputs of other accelerators (such as accelerator b 142). Therefore, after receiving a first inference request, accelerator a 141 can distribute the first inference request to accelerator b 142.

[0170] Similarly, in another possible example, accelerator b 142 is the main accelerator in the model inference system 140. Therefore, after receiving the first inference request, accelerator b 142 can distribute the first inference request to accelerator b 141.

[0171] In yet another possible example, both accelerator a 141 and accelerator b 142 directly receive the first inference request.

[0172] For example, accelerator a 141 and accelerator b 142 receive the first inference request distributed by processor 140a.

[0173] S320 and accelerator a 141 process the first inference request according to the first thermal weight and obtain the first inference result.

[0174] Among them, the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0175] The activation frequency mentioned above refers to the frequency of weight usage in the model during offline prediction or inference. For example, when the model processes different input data multiple times, the processor 140a counts the weights used in each processing of the input data, and then uses the ratio of the number of times each weight is used to the total number of times the model processes the input data as the activation frequency.

[0176] Furthermore, the processor 140a can use the activation frequencies of the aforementioned weights to determine the hot weights and cold weights (which can be referred to as hot and cold weights) in the model.

[0177] For processor 140a, details on how to determine the hot and cold weights in the model using the activation frequencies of the aforementioned weights can be found below. Figure 5 The specific wording is omitted here.

[0178] In one possible example, the activation frequency of the first hot weight is greater than or equal to the threshold a, and the activation frequency of the cold weight in the model is less than the threshold a, so that the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0179] In another possible example, the activation frequency of the first hot weight is greater than the threshold a, and the activation frequency of the cold weight in the model is less than or equal to the threshold a, so that the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0180] The threshold 'a' mentioned above can be configured by the user.

[0181] In one possible scenario, the first hot weight is the hot weight in the weight matrix corresponding to one or more network layers in the model.

[0182] For example, the first hot weight is the set of hot weights in the weight matrix corresponding to each network layer in the model.

[0183] In one possible implementation, accelerator a 141 processes the first inference request according to the first hot weight to obtain the first inference result, including: accelerator a 141 processes the sequence in the first inference request according to the first hot weight to obtain the first inference result.

[0184] In one possible example, accelerator a 141 multiplies the first heat weights with the vector corresponding to the sequence to obtain the first inference result. This first inference result is also a vector.

[0185] In one possible implementation, since processor 140a obtains the weights required by multiple accelerators in the model inference system to process the first inference request, and the aforementioned required weights include the first sub-hot weights in the first hot weights, then accelerator a 141 processes the first inference request according to the first hot weights to obtain the first inference result, including: accelerator a 141 processes the first inference request according to the first sub-hot weights to obtain the first inference result.

[0186] For example, accelerator a 141 multiplies the first sub-thermal weights with the sequence to obtain the first inference result.

[0187] For more possible implementations of S320, please refer to the following: Figure 6 The specific wording is omitted here.

[0188] It is worth noting that the aforementioned first hot weight can be stored in the storage space of the data processing system 100. This storage space can be the level 1 cache (L1), level 2 cache (L2), level 3 cache (L3), high bandwidth memory (HBM), etc. of the accelerator a 141, or the storage space can be located in a memory connected to the first accelerator.

[0189] S330 and accelerator b 142 process the first inference request according to the second thermal weight to obtain the second inference result.

[0190] In one possible scenario, the first thermal weight and the second thermal weight are the same.

[0191] In one possible example, the second hot weights are the hot weights in the weight matrix corresponding to one or more network layers in the model.

[0192] For example, the second hot weight is the set of hot weights in the weight matrix corresponding to each network layer in the model.

[0193] For more details on the second heat weight, please refer to the description of the first heat weight above, which will not be repeated here.

[0194] In one possible implementation, accelerator b 142 processes the first inference request according to the second hot weight to obtain the second inference result, including: accelerator a 141 processes the sequence in the first inference request according to the second hot weight to obtain the second inference result.

[0195] In one possible example, accelerator b 142 multiplies the second heat weights by the vector corresponding to the sequence to obtain a second inference result. This second inference result is also a vector.

[0196] In one possible implementation, since processor 140a can obtain the weights required for multiple accelerators in the model inference system to process the first inference request, and the aforementioned required weights include the second sub-hot weights in the second hot weights, accelerator b 142 processes the first inference request according to the second hot weights to obtain the second inference result, including: accelerator a 141 processes the first inference request according to the second sub-hot weights to obtain the second inference result.

[0197] For example, accelerator a 141 multiplies the second sub-heat weight with the vector corresponding to the sequence to obtain the second inference result.

[0198] For more possible implementations of the S330, please refer to the following: Figure 6 The specific wording is omitted here.

[0199] It is worth noting that the aforementioned second thermal weight can be stored in the storage space of accelerator b 142, which can be L1, L2, L3, HBM, etc. of the first accelerator.

[0200] S340 and accelerator a 141 obtain the output results based on the first and second inference results.

[0201] In this embodiment, accelerator a 141 is used as the main accelerator for illustration. After receiving the second inference result sent by accelerator b 142, accelerator a 141 merges the first inference result with the second inference result to obtain the output result.

[0202] In one possible example, if the computation of the first inference request by accelerator a 141 using the first hot weight is completely independent of the computation of the second inference request by accelerator b 142 using the second hot weight, then accelerator a 141 will concatenate the first inference result and the second inference node.

[0203] For example, if the first inference result is a 2x3 matrix a and the second inference result is a 2x3 matrix b, then accelerator a141 will concatenate matrices a and b to obtain a 4x3 matrix c.

[0204] In another possible example, if the computation involved in the first network layer corresponding to the first hot weight is complex, then accelerator a 141 needs to perform further processing when merging the first inference result with the second inference result, such as involving weighted sums, activation functions, etc.

[0205] For example, the first inference result is a 2x3 matrix a, and the second inference result is a 2x3 matrix b. Accelerator a 141 weights matrices a and b and then concatenates them to obtain a 4x3 matrix d. The values ​​in matrix d are different from the values ​​in the matrix c mentioned above.

[0206] For example, the first inference result is a 2x3 matrix a, and the second inference result is a 2x3 matrix b. Accelerator a141 concatenates matrices a and b to obtain a 4x3 matrix c, and then uses an activation function to process matrix c to obtain the final output result.

[0207] In one possible scenario, accelerator a 141 sends the obtained output to terminal device 150.

[0208] In other embodiments of this application, the main accelerator in the model inference system 140 may be accelerator b 142, and then accelerator b 142 obtains the output result based on the first inference result and the second inference result.

[0209] Regarding the determination of hot and cold weights in the model by the processor 140a in the above S320, the following is a possible embodiment.

[0210] The processor 140a obtains the weights required by the model inference system 140 when processing historical inference requests, and then obtains the activation frequency of each weight in the model based on the weights required for each of the multiple requests included in the historical inference requests.

[0211] In one possible example, such as Figure 4 As shown, Figure 4 This application provides a schematic diagram for determining hot and cold weights. The historical inference requests include inference request 1, inference request 2, and inference request 3. When the model processes inference request 1, the weights required are weight a, weight b, and weight c. When the model processes inference request 2, the weights required are weight a, weight c, and weight d. When the model processes inference request 3, the weights required are weight a, weight d, and weight e.

[0212] Processor 140a calculates the activation frequency of each weight based on the weights required for the aforementioned inference requests. For example, the activation frequency of weight a is the number of times weight a is activated (3 times) divided by the total number of requests included in the historical inference requests (3 times), which equals 100%. Similarly, the activation frequency of weight b is 1 / 3, or 33%, the activation frequency of weight c is 2 / 3, or 67%, the activation frequency of weight d is 2 / 3, or 67%, and the activation frequency of weight e is 1 / 3, or 33%.

[0213] Furthermore, the processor 140a, based on the user-configured threshold a (e.g., 50%), designates the weights with activation frequencies greater than or equal to threshold a as hot weights, such as weight a, weight c, and weight d; and designates the weights with activation frequencies greater than or equal to threshold a as cold weights, such as weight b and weight e.

[0214] In one possible scenario, processor 140a may also determine the co-occurrence of weights based on the weights required by the model when processing historical inference requests.

[0215] The co-occurrence mentioned above generally refers to the frequency with which different weights are jointly activated or used across different inputs to the model. Co-occurrence can be used to understand the relationships between weights and how they collectively affect the model's output.

[0216] In one possible example, processor 140a determines the weights that are simultaneously activated during the reasoning process of each reasoning request in the history of reasoning requests, and then identifies the co-occurrence between the weights through frequent item mining, and stores the weights with a co-occurrence greater than or equal to a threshold b (a second threshold) in the same storage unit.

[0217] The aforementioned weight combinations may include at least two weights, and multiple weight combinations may be formed from all the weights in the model. The aforementioned storage units may be blocks, cache lines, segments, or chunks in the accelerator, or blocks, cache lines, segments, or chunks in memory connected to the accelerator.

[0218] In one possible example, processor 140a can determine the co-occurrence of weights by performing frequent item mining using algorithms such as Apriori and Frequent Pattern Growth (FP-Growth).

[0219] It is worth noting that the processor 140a can determine the hot and cold weights in the model, and this can also be performed by the computing device 110.

[0220] The above describes how processor 140a switches between hot and cold weights based on the weights required to be used corresponding to historical inference requests. The following also provides an embodiment of a model training system 120 training an activation classifier based on historical inference requests and their corresponding required weights. The activation classifier can be used to process a first inference request to obtain the weights required by the model inference system 140 when processing the first inference request, thereby achieving the aforementioned flyline prediction.

[0221] The data processing system 100 trains an activation classifier using historical inference requests and their corresponding weights. This trained activation classifier establishes a mapping relationship between inference requests and the weights to be used.

[0222] In one possible scenario, the activation classifier mentioned above could be logistic regression, support vector machine (SVM), decision tree, random forest, neural network (such as multi-layer perceptron (MLP), convolutional neural network (CNN), etc.).

[0223] The aforementioned historical inference requests, and the weights required by the model to process these requests, can be obtained by the model training system 120 from the database 130. The historical inference requests and their corresponding weights stored in the database 130 can be written by the terminal device 150, or collected and written by the data acquisition device 170.

[0224] like Figure 5 As shown, Figure 5This is a schematic diagram illustrating the training of an activation classifier provided in this application. The model training system 120 trains the activation classifier using historical inference requests and their corresponding weights. The activation classifier obtained through the aforementioned training can actually establish a mapping relationship between the features of the inference request and the weights to be used.

[0225] The weights required to be used include their positions in the weight matrix.

[0226] For example, the weights required above represent the weights that the model actually needs to use when processing historical inference requests. For instance, when the model processes historical inference request a, it needs to use weight a from the full weight set; when the model processes historical inference request b, it needs to use weight b from the full weight set; and when the model processes historical inference request c, it needs to use weight c from the full weight set.

[0227] The training and activation of the classifier described above can be referred to conventional techniques in this field, and will not be elaborated upon here.

[0228] It is worth noting that the processor 140a described above determines the hot and cold weights in the model and the content of training the activation classifier as an offline preparation process before using the model for inference.

[0229] Regarding the processor 140a mentioned in S310 above performing flyline prediction on the first inference request and thereby determining the weights to be used when multiple accelerators in the model inference system process the first inference request, the following is a possible embodiment.

[0230] The processor 140a inputs the first inference request into the activation classifier obtained after training, outputs the weights required by the model to process the first inference request, and then distributes the weights required to process the first inference request to multiple accelerators.

[0231] In one possible example, processor 140a sends the identifier of the weights that each accelerator needs to utilize to the corresponding accelerator.

[0232] For example, processor 140a sends the identifiers of the weights that accelerator a 141 needs to utilize to accelerator a 141. For example, the aforementioned identifiers include the storage address of weight a, the storage address of weight b, and the first hot weights include weight a and weight d.

[0233] For example, the position of weight a in the weight matrix corresponding to the first hot weight, such as (1, 2); the position of weight d in the weight matrix corresponding to the first hot weight, such as (2, 3).

[0234] Processor 140a sends the identifiers of the weights required by accelerator b 142 to accelerator b 142. For example, the aforementioned identifiers include the storage address of weight e, the storage address of weight f, and the second hot weights include weight e and weight f.

[0235] For example, the position of weight e in the weight matrix corresponding to the first hot weight, such as (1, 2); the position of weight f in the weight matrix corresponding to the first hot weight, such as (2, 3).

[0236] Regarding the content of S320 described above, the following is a possible embodiment. In this embodiment, as... Figure 6 As shown, Figure 6 A schematic diagram of a hot and cold weight configuration provided in this application Figure 1 Accelerator a 141 has a first hot weight and a first cold weight, and accelerator b 142 is configured with a second hot weight and a second cold weight.

[0237] like Figure 6 As shown, since accelerator a 141 and accelerator b 142 are of the same type, processor 140a assigns the same hot weights (first hot weight and second hot weight) to accelerator a 141 and accelerator b 142. Furthermore, to avoid excessive storage resource consumption by each accelerator, processor 140a divides the aforementioned determined cold weights into multiple parts and stores them in different accelerators; that is, one cold weight is stored in each accelerator. For example, processor 140a assigns the first cold weight to accelerator a 141 and the second cold weight to accelerator b 142.

[0238] In one possible scenario, the aforementioned accelerator a 141 and accelerator b 142 are of the same type, used to indicate that accelerator a 141 and accelerator b 142 are identical in one or more of the following: equipment manufacturer, product model, performance, etc.

[0239] In one possible example, processor 140a copies thermal weights to accelerator a 141 and accelerator b 142 via broadcast. The first thermal weight can be stored in the HBM of accelerator a 141, and the second thermal weight can be stored in the HBM of accelerator b 142.

[0240] For example, weight combinations with high co-occurrence in the first hot weights can be stored in the same storage unit in the HBM of accelerator a 141, such as the same block, the same cache line, the same data segment, or the same chunk, thereby improving the efficiency of accelerator a 141 in reading weights in the HBM.

[0241] Similarly, weight combinations with high co-occurrence in the second hot weights can be stored in the same storage unit in the HBM of accelerator b 142, such as the same block, the same cache line, the same data segment, or the same chunk, thereby improving the efficiency of accelerator b 142 in reading weights from the HBM.

[0242] For example, weight combinations with low co-occurrence in the first hot weights can be stored in the same storage cell in the HBM of accelerator a 141, and weight combinations with low co-occurrence in the second hot weights can be stored in the same storage cell in the HBM of accelerator b 142.

[0243] Alternatively, the weight combinations with low co-occurrence in the first hot weights can be stored in the HBM of accelerator a 141, and the weight combinations with low co-occurrence in the second hot weights can be stored in the HBM of accelerator b 142.

[0244] For example, weight combinations with low co-occurrence in the first hot weights can be stored in different memory cells in the HBM of accelerator a 141, and weight combinations with low co-occurrence in the second hot weights can be stored in different memory cells in the HBM of accelerator b 142.

[0245] It is worth noting that the above-mentioned high co-occurrence is used to indicate that the co-occurrence is greater than or equal to the threshold b, and low co-occurrence is used to indicate that the co-occurrence is less than the threshold b.

[0246] It is worth noting that the above description uses the example of accelerator a 141 and accelerator b 142 being of the same type. In other embodiments of this application, accelerator a 141 and accelerator b 142 may be of different types, and the first thermal weight configured in accelerator a 141 and the second thermal weight configured in accelerator b 142 may still be the same.

[0247] like Figure 7 As shown, Figure 7 A schematic diagram of a hot and cold weight configuration provided in this application Figure 2 Accelerator a 141 and accelerator b 142 are configured with the same thermal weights, and accelerator a 141 and accelerator b 142 are configured with different cold weights.

[0248] For example, processor 140a divides the cold weights in the model into multiple parts using tensor parallelism. Then, processor 140a distributes one set of cold weights to each accelerator. For instance, processor 140a configures a first set of cold weights for accelerator a 141 and a second set of cold weights for accelerator b 142.

[0249] For example, processor 140a vertically or horizontally splits the cold weights of each network layer in the model, and then configures the set of weights obtained after splitting each network layer to the accelerator.

[0250] This application does not limit the number of times the data is split vertically or horizontally. The splitting can be performed according to the number of accelerators so that the number of weighted components obtained after the split matches the number of processors.

[0251] In one possible scenario, the first cold weights include a weight combination 'a' with high co-occurrence, and the second cold weights include a weight combination 'b' with high co-occurrence. The weight combination 'a' with high co-occurrence in the first cold weights is stored in the same storage unit in accelerator a 141, and the weight combination 'b' with high co-occurrence in the second cold weights is stored in the same storage unit in accelerator b 142.

[0252] This application does not limit the number of weight combinations included in the first cold weight and the second cold weight.

[0253] In one possible scenario, the multiple accelerators included in the model inference system 140 are configured with the same thermal weights.

[0254] It is worth noting that, to ensure load balancing across accelerators, the difference in data volume between the first cold weight and the second cold weight is less than or equal to a threshold c, which can be configured by the user. In one possible example, the data volume of the first cold weight and the second cold weight are the same.

[0255] Regarding the details of S320, two possible implementation methods are provided below.

[0256] In one possible implementation, accelerator a 141 obtains all the first hot weights and all the first cold weights from the storage space within accelerator a 141, and then uses all the first hot weights and all the first cold weights to process the first inference request and obtain the first inference result.

[0257] The storage space within the aforementioned accelerator a 141 can be L1, L2, L3, HBM, etc. in the accelerator a 141.

[0258] For example, accelerator a 141 multiplies all the first hot weights and all the first cold weights with the sequence in the first inference request to obtain the first inference result.

[0259] Similarly, accelerator b 142 obtains all the second hot weights and all the second cold weights from the storage space within accelerator b 142, and then uses all the second hot weights and all the second cold weights to process the first inference request and obtain the second inference result.

[0260] For details regarding the second inference result obtained by accelerator b 142, please refer to the description of the first inference result obtained by accelerator a 141 in this implementation, which will not be repeated here.

[0261] In another possible implementation, accelerator a 141 obtains the first sub-hot weight in the first hot weight and the first sub-cold weight in the first cold weight from the storage space within accelerator a 141, and then uses the first sub-hot weight and the first sub-cold weight to process the first inference request and obtain the first inference result.

[0262] For example, accelerator a 141 multiplies the first sub-hot weight and the first sub-cold weight with the sequence to obtain the first inference result.

[0263] It is worth noting that the activation classifier in processor 140a processes the first inference request to obtain weights required by multiple accelerators when processing the first inference request. These weights include first sub-hot weights and first sub-cold weights. Then, processor 140a determines, from the correspondence between weights and accelerators, that the first sub-hot weights and first sub-cold weights are deployed in accelerator a 141. Therefore, processor 140a sends the identifiers of the first sub-hot weights and the first sub-cold weights to accelerator a 141, and accelerator a 141 retrieves the first sub-hot weights and the first sub-cold weights from its storage space based on these identifiers.

[0264] Similarly, accelerator b 142 obtains the second sub-hot weight and the second sub-cold weight from the storage space within accelerator b 142, and uses the second sub-hot weight and the second sub-cold weight to process the first inference request to obtain the second inference result.

[0265] For details regarding the second inference result obtained by accelerator b 142, please refer to the description of the first inference result obtained by accelerator a 141 in this implementation, which will not be repeated here.

[0266] To achieve load balancing between accelerator a 141 and accelerator b 142, the data for the first sub-hot weight and the first sub-cold weight acquired by accelerator a 141 should be substantially the same as the data for the second sub-hot weight and the second sub-cold weight acquired by accelerator b 142. This ensures that accelerator a 141 and accelerator b 142 have the same computational load and therefore the same overall load. Figure 8 As shown, Figure 8 The present application provides a load balancing distribution diagram. The processor 140a will use a load balancing algorithm to determine the weight required by each accelerator when processing the first inference request, and then send the identifier of the weight required by each accelerator to the corresponding accelerator.

[0267] The processor 140a determines the weights required for the model to process the input data included in the first inference request by activating the classifier, and then inputs the required weights into the equalizer. The equalizer determines, through a load balancing algorithm, the weights required for the accelerator a141 to process the first inference request, including the first sub-hot weights and the first sub-cold weights, and the weights required for the accelerator b142 to process the first inference request, including the second sub-hot weights and the second sub-cold weights.

[0268] The difference between the sum of the data volume of the first sub-hot weight and the data volume of the first sub-cold weight and the sum of the data volume of the second sub-hot weight and the data volume of the second sub-cold weight is less than or equal to the threshold d (the first threshold), thereby achieving load balancing between accelerator a 141 and accelerator b 142 during model inference. Figure 8 Each box contains the same amount of weight data.

[0269] For example, processor 140a determines the weights required for the model to process the first inference request by activating the classifier, including 4 hot weights and 8 cold weights, wherein the 4 cold weights are configured in accelerator a 141 and the 4 cold weights are configured in accelerator b 142.

[0270] Therefore, to ensure load balancing between accelerators a 141 and b 142, the equalizer allocates computationally similar data to accelerators a 141 and b 142 for calculation (e.g., the difference between the sum of the data amounts of the first sub-hot weight and the first sub-cold weight, and the sum of the data amounts of the second sub-hot weight and the second sub-cold weight, is less than or equal to a threshold d). Since the hot weights configured in accelerators a 141 and b 142 are consistent, processor 140a can instruct accelerator a 141 to calculate the first inference request using the aforementioned 2 hot weights (first sub-hot weights) and 4 cold weights (first sub-cold weights), and instruct accelerator b 142 to calculate the first inference request using the aforementioned 2 hot weights (second sub-hot weights) and 4 cold weights (second sub-cold weights).

[0271] For example, processor 140a determines the weights required for the model to process the first inference request by activating the classifier, including 6 hot weights and 6 cold weights, of which 2 cold weights are configured in accelerator a 141 and 4 cold weights are configured in accelerator b 142.

[0272] Therefore, processor 140a can instruct accelerator a 141 to calculate the first inference request using the aforementioned 4 hot weights (first sub-hot weights) and 2 cold weights (first sub-cold weights), and instruct accelerator b 142 to calculate the first inference request using the aforementioned 2 hot weights (second sub-hot weights) and 4 cold weights (second sub-cold weights).

[0273] In the example above, one weight can be: Figure 8 The weight corresponding to a box in the middle.

[0274] The equalizer mentioned above can be deployed in processor 140a.

[0275] In one possible embodiment, in Figure 3 Based on the model reasoning method shown, this application also provides a method for switching between hot and cold weights. Figure 9 This is a flowchart illustrating a hot / cold weight switching method provided in this application. In this embodiment, the processor 140a maintains a counting matrix, which is used to indicate the activation frequency of weights in the model. Figure 9 As shown, the hot and cold weight switching method may include the following steps S910 to S940.

[0276] S910, Processor 140a determines the first position of the weights required by model inference system 140 when processing the first inference request in the weight matrix; In one possible implementation, processor 140a determines the first position of the weights that the model inference system 140 needs to utilize in each inference request in the weight matrix.

[0277] The aforementioned reasoning request includes the first reasoning request.

[0278] In one possible scenario, the first position is the position of the weight to be used in the weight matrix, and this first position can be used to determine the second position in the counting matrix corresponding to the activation frequency of the weight.

[0279] In one possible example, processor 140a can process the inference request by activating the classifier to obtain the first position of the weights to be used in the weight matrix.

[0280] S920, processor 140a increments the value at the second position in the counting matrix corresponding to the first position by 1 according to the first position.

[0281] In one possible scenario, the tracer in processor 140a maintains a corresponding count matrix for each weight matrix in the model.

[0282] In one possible example, the size of the counting matrix is ​​the same as the size of the weight matrix, and the first position is the specific position of the weight to be used in the weight matrix. Then, after determining the first position, the processor 140a increments the value at the second position in the counting matrix corresponding to the first position by 1.

[0283] For example, both the counting matrix and the weight matrix are 5x5 matrices. Therefore, the positions in the counting matrix correspond one-to-one with the positions in the weight matrix. If the weight to be used includes weight a, and the first position of weight a in the weight matrix is ​​(3, 4) in the third row and fourth column, the second position is the same as the first position, which is also (3, 4). Therefore, the processor 140a increments the value at (3, 4) in the counting matrix by 1.

[0284] In another possible example, the number of columns in the counting matrix is ​​the same as the number of columns in the weight matrix, the first position is the column of the weight to be used in the weight matrix, and the counting matrix has only one row. Then, after determining the first position, the processor 140a increments the value at the second position in the counting matrix corresponding to the first position by 1.

[0285] For example, the weight matrix is ​​a 5x5 matrix, and the counting matrix is ​​a 1x5 matrix. Therefore, a column in the weight matrix corresponds to a position in the weight matrix. If the weight to be used includes weight a, and the first position of weight a in the weight matrix is ​​(3, 4), the second position has the same column number as the first position. Therefore, the second position is (1, 4), and the processor 140a increments the value at (1, 4) in the counting matrix by 1.

[0286] In another possible example, the number of columns in the counting matrix is ​​the same as the number of rows in the weight matrix, the first position is the row of the weight to be used in the weight matrix, and the counting matrix has only one row. Then, after determining the first position, the processor 140a increments the value at the second position in the counting matrix corresponding to the first position by 1.

[0287] For example, the weight matrix is ​​a 5x5 matrix, and the counting matrix is ​​a 1x5 matrix. Therefore, a row in the weight matrix corresponds to a position in the weight matrix. For instance, if the weights to be used include weight a, and weight a is located at (3, 4) in the first position of the weight matrix, the number of rows at the first position is the same as the number of columns at the second position. Therefore, the second position is (1, 3), and the processor 140a increments the value at (1, 3) in the counting matrix by 1.

[0288] S930. If the value at the third position in the counting matrix is ​​greater than or equal to the threshold e (third threshold), then the processor 140a sends a first weight switching request to the accelerator a 141.

[0289] The first weight switching request is used to instruct the switching of the first cold weight configured in the first accelerator.

[0290] The weight corresponding to the third position mentioned above is configured in accelerator a 141. Therefore, when the trigger (tracker) in processor 140a determines that the value at the fourth position in the counting matrix is ​​greater than or equal to the threshold e, it sends a first weight switching request to accelerator a 141.

[0291] In one possible example, if the counting matrix and the weight matrix are of the same size, then the weight at the third position is a single weight. For example, if the third position is (3, 4), then the weight at the third position is the weight at position (3, 4) in the weight matrix.

[0292] In another possible example, if the counting matrix and the weight matrix have the same number of columns, then the weight at the third position is a column weight. For example, if the third position is (1, 2), then the weight at the third position is the weight corresponding to the second column in the weight matrix.

[0293] In another possible example, if the number of columns in the counting matrix is ​​the same as the number of rows in the weight matrix, then the weight at the third position is the weight of a row. For example, if the third position is (1, 2), then the weight at the third position is the weight corresponding to the second row in the weight matrix.

[0294] In one possible implementation, if the data at the third position in the counting matrix is ​​greater than or equal to a threshold e (e.g., 50), the processor 140a sends a first weight switching request to the accelerator a 141 to indicate the switching of the third sub-cold weights (including the aforementioned single weight, column weight, and row weight).

[0295] If the data at the third position is greater than or equal to the threshold e, it means that the activation frequency of the third sub-cold weight corresponding to the third position is greater than the activation frequency of other weights in the cold weights except for the third sub-cold weight.

[0296] It is worth noting that the above description only uses the weight corresponding to the third position as an example of being configured in accelerator a 141. In other embodiments of this application, the weight corresponding to the third position can also be configured in other accelerators in the model inference system 140.

[0297] S940 and accelerator a 141 switch the third sub-cold weight in the first cold weight to a hot weight according to the first weight switching request.

[0298] Among them, the activation frequency of the third sub-cold weight is greater than the activation frequency of the other weights in the cold weights except for the third sub-cold weight.

[0299] In one possible scenario, the distance between the storage area of ​​the first hot weight and the acceleration chip in the first accelerator is less than or equal to the distance between the storage area of ​​the first cold weight and the aforementioned acceleration chip. Therefore, according to the first weight switching request, the accelerator a 141 migrates the third sub-cold weight from the storage area of ​​the first cold weight to the storage area of ​​the first hot weight, thereby realizing the use of the third sub-cold weight as a hot weight.

[0300] In one possible example, the storage areas of the first hot weight and the first cold weight are both located in the HBM of accelerator a 141. Accelerator a 141 can copy the third sub-cold weight to the temporary space (buffer) of the HBM, and then copy the third sub-cold weight in the aforementioned buffer to the storage area of ​​the first hot weight, thereby realizing the migration of the third sub-cold weight from the storage area of ​​the first cold weight to the storage area of ​​the first hot weight, thus realizing the online migration of hot and cold weights.

[0301] In one possible scenario, after accelerator a 141 switches the third sub-cold weight in the first cold weight to a hot weight, it sends a switching completion response to processor 140a. Based on the aforementioned response, processor 140a restores the counting state of the counting matrix maintained by the tracker, that is, clears the counting matrix to zero.

[0302] It is worth noting that since the multiple accelerators included in the model inference system 140 are configured with the same hot weights, when accelerator a 141 performs a cold weight switch, the other accelerators in the model inference system 140 also need to perform a hot weight switch. For example... Figure 9 As shown, since the third sub-cold weight is stored in the storage space within accelerator a 141, the third sub-cold weight can be transmitted between accelerators via high-speed interconnect bandwidth, so that other accelerators can use the third sub-cold weight as a hot weight.

[0303] Taking accelerator b 142 in other accelerators as an example, and using HBM where the third sub-cold weight is stored in accelerator a 141, accelerator a 141 copies the third sub-cold weight to the HBM buffer, and synchronizes the third sub-cold weight globally (all accelerators) through the high-speed interconnect bandwidth between accelerators. Then, accelerator b 142 switches the third sub-cold weight to a hot weight.

[0304] For example, accelerator a 141 uses high-speed interconnect bandwidth to perform ALL2ALL communication to synchronize data (third sub-cold weight) globally, and the aforementioned data exchange is completed in the buffer, realizing online switching of cold and hot weights.

[0305] Accelerator a 141 synchronizes the third sub-cold weight stored in the buffer of accelerator a 141 to the buffer of accelerator b 142 through the high-speed interconnect bandwidth between accelerator a 141 and accelerator b 142. Then, accelerator b 142 can store the third sub-cold weight in its buffer to the storage area corresponding to the first hot weight, so that accelerator b 142 can use the third sub-cold weight as a hot weight.

[0306] For example, the high-speed interconnect bandwidth between the aforementioned accelerators could be RDMA (remote direct memory access), which can include: IB (infiniband), RoCE (RDMA over converged Ethernet), iWARP (internet wide area RDMA protocol), etc. IB is a high-performance interconnect technology that provides native RDMA support. RoCE is a technology that implements RDMA over Ethernet. It uses standard Ethernet as the underlying transmission medium and implements RDMA functionality through the use of RoCE adapters and appropriate protocol stacks. iWARP is an RDMA implementation based on the TCP / IP protocol stack. It uses ordinary Ethernet adapters and standard network switches, and provides high-performance remote memory access and data transmission by implementing RDMA functionality within the TCP / IP protocol stack.

[0307] It is worth noting that the above content should not be construed as limiting this application. In another embodiment of this application, if the value at the fourth position in the counting matrix is ​​less than the threshold f (the fourth threshold), the processor 140a sends a second weight switching request to the accelerator a 141. According to the second weight switching request, the accelerator a 141 switches the third sub-hot weight in the first hot weights to a cold weight. The second switching request is used to instruct the switching of the first hot weights configured in the accelerator a 141, and the activation frequency of the third sub-hot weight is less than the activation frequency of the other weights in the first hot weights besides the third sub-hot weight.

[0308] The contents of the counting matrix can be referred to the description in the above embodiment of hot and cold weight switching, and will not be repeated here.

[0309] The weight corresponding to the fourth position is the third sub-hot weight (single weight, column weight, row weight). The value at the fourth position is less than the threshold f, which means that the activation frequency of the third sub-hot weight corresponding to the fourth position is less than the activation frequency of the other weights in the first hot weight except for the third sub-hot weight.

[0310] In one possible scenario, the distance between the storage area of ​​the first hot weight and the acceleration chip in the first accelerator is less than or equal to the distance between the storage area of ​​the first cold weight and the aforementioned acceleration chip. Therefore, according to the second weight switching request, the accelerator a 141 migrates the third sub-hot weight from the storage area of ​​the first hot weight to the storage area of ​​the first cold weight, thereby realizing the switching of the third sub-hot weight to a cold weight.

[0311] In one possible example, the storage area of ​​the first hot weight and the storage area of ​​the first cold weight are both located in the HBM in accelerator a 141. Accelerator a 141 can copy the third sub-hot weight to the buffer of the HBM, and then copy the third sub-hot weight in the aforementioned buffer to the storage area of ​​the first cold weight, so as to realize the migration of the third sub-hot weight from the storage area of ​​the first hot weight to the storage area of ​​the first cold weight, thereby realizing the online migration of hot and cold weights.

[0312] It is worth noting that since the multiple accelerators included in the model inference system 140 are configured with the same hot weights, when accelerator a 141 performs a cold weight switch, the other accelerators in the model inference system 140 also need to perform a hot-cold weight switch. The details of the hot-cold weight switch for other accelerators can be found in the embodiments described above, and will not be repeated here.

[0313] In another embodiment of this application, if the value at the third position in the counting matrix is ​​greater than threshold e, and the value at the fourth position in the counting matrix is ​​less than threshold f, a third weight switching request is sent to accelerator a 141. Accelerator a 141, according to the third weight switching request, switches the third sub-hot weight in the first hot weights to a cold weight, and switches the third sub-cold weight in the first cold weights configured in the first accelerator to a hot weight. The third weight switching request is used to indicate the switching of weights configured in the first accelerator. The activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weights except for the third sub-hot weight, and the activation frequency of the third sub-cold weight is greater than the activation frequency of other weights in the cold weights except for the third sub-cold weight.

[0314] For details of this embodiment, please refer to the description of the two possible embodiments for switching between hot and cold weights, which will not be repeated here.

[0315] To fully and clearly illustrate the overall process of the model inference method, a possible embodiment is provided below. For example... Figure 10 As shown, Figure 10 A flowchart illustrating a model reasoning method provided in this application. Figure 2 . Figure 10 The method shown includes the following steps S1010-S1050.

[0316] S1010 and model training system 120 train the activation classifier.

[0317] For details regarding S1010, please refer to the above. Figure 5 The content shown will not be repeated here.

[0318] S1020, the model inference system 140 determines the hot and cold weights in the model and deploys the same hot weights and different cold weights for each accelerator.

[0319] For details regarding S1020, please refer to the above. Figure 4 , Figure 6 , Figure 7 The content shown will not be repeated here.

[0320] S1030 and the model inference system 140 use an activation classifier to determine the weights required for multiple accelerators to process the first inference request, and the multiple accelerators process the first inference request according to the required weights to obtain the output result.

[0321] For details regarding S1030, please refer to the above. Figure 3 , Figure 8 The content shown will not be repeated here.

[0322] S1040, during the process of processing the first inference request, the model inference system 140 calculates the activation frequency / number of activations of the weights.

[0323] S1050 and model inference system 140 switch between hot and cold weights in the model based on the activation frequency / number of activations of the weights.

[0324] For details regarding S1040 and S1050, please refer to the above. Figure 9 The content shown will not be repeated here.

[0325] The foregoing describes a homogeneous model inference system. In one possible embodiment, the model inference system can also be heterogeneous. For example, the model inference system 140 may include a third accelerator among the plurality of accelerators, the third accelerator being of a different type from the first and second accelerators.

[0326] In one possible scenario, the third accelerator is configured with a third hot weight for the model, which is different from the first hot weight.

[0327] In this case, the heat weights in the model are divided into multiple parts, such as two parts, one of which is the first heat weight or the second heat weight, and the other is the third heat weight.

[0328] For example, the hot weights corresponding to each network layer in the model are divided into two parts: the first hot weights include one part of the hot weights corresponding to each network layer in the model, and the third hot weights include another part of the hot weights corresponding to each network layer in the model.

[0329] In one possible implementation, the above model inference method further includes: a third accelerator processing the first inference request according to a third hot weight to obtain a third inference result. Furthermore, the first accelerator obtains an output result based on the first and second inference results, including: the first accelerator obtaining an output result based on the first, second, and third inference results.

[0330] The content of the output result obtained by the first accelerator based on the first inference result, the second inference result, and the third inference result can be referred to the above description of the output result obtained by the first accelerator based on the first inference result and the second inference result, and will not be repeated here.

[0331] In another possible scenario, the third accelerator is configured with the model's third hot weights, which are the same as the first hot weights.

[0332] In this case, there is only one set of hot weights in the model, and each accelerator stores the same set of hot weights.

[0333] For details regarding this situation, please refer to the above-mentioned situations; they will not be repeated here.

[0334] The above illustrates an embodiment where the first thermal weight and the second thermal weight are the same. In one possible embodiment, the first thermal weight and the second thermal weight may be different.

[0335] That is, the heat weights in the model are divided into multiple parts, such as two parts, one of which is the first heat weight and the other is the second heat weight.

[0336] For example, the hot weights corresponding to each network layer in the model are divided into two parts: the first hot weights include one part of the hot weights corresponding to each network layer in the model, and the second hot weights include another part of the hot weights corresponding to each network layer in the model.

[0337] For the content of this embodiment, please refer to the description of the embodiment above where the first thermal weight and the second thermal weight are the same, and it will not be repeated here.

[0338] It is understood that, in order to achieve the functions described in the above embodiments, the model inference system includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0339] This application embodiment also provides a chip, which includes a control circuit and an interface circuit. The interface circuit is used to obtain a first inference request, and the control circuit is used to implement the function of accelerator a 141 or accelerator b 142 in the above method.

[0340] In one possible design, the chip also includes a memory for storing program instructions and / or data.

[0341] For details regarding the implementation of the chip provided in the embodiments of this application, please refer to the functions of accelerator a 141 or accelerator b 142 described above, which will not be repeated here.

[0342] This application also provides a server, which includes a processor, a plurality of accelerators, and a memory. The plurality of accelerators includes a first accelerator and a second accelerator. The processor is used to execute instructions stored in the memory to obtain a first inference request. The first accelerator is used to implement the function of accelerator a 141 in the above method embodiment, and the second accelerator is used to implement the function of accelerator b 142 in the above method embodiment.

[0343] In other words, the server provided in this application embodiment is a supernode. A supernode refers to multiple GPUs interconnected via a bus, which appears as a larger server.

[0344] For details regarding the implementation of the chip provided in this application embodiment, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0345] This application also provides an accelerator. The accelerator provided in this application embodiment will be described below with reference to the accompanying drawings. Figure 11 This is a schematic diagram of an accelerator structure provided in this application. The accelerator 1100 can be used to implement the functions of the accelerator in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the accelerator 1100 can be as follows: Figure 1 The accelerator shown is a 141, also known as the first accelerator.

[0346] like Figure 11As shown, accelerator 1100 may include accelerator chip 1120. Optionally, accelerator 1100 may also include memory 1130 and / or transceiver 1110. Accelerator chip 1120 is coupled to memory 1130 and transceiver 1110, for example, via a communication bus. This communication bus may include, but is not limited to: PCIe bus, or extended industry standard architecture (EISA) bus, unified bus (Ubus or UB), compute express link (CXL), cache coherent interconnect for accelerators (CCIX), etc.

[0347] The following is combined Figure 11 A detailed introduction to each component of the accelerator 1100: The acceleration chip 1120 is the computing unit of the accelerator 1100, and can be a control circuit or a collective term for multiple processing elements. For example, the acceleration chip 1120 can be one or more GPUs, TPUs, NPUs, or ASICs, or one or more integrated circuits configured to implement the embodiments of this application, such as one or more DSPs, or one or more FPGAs.

[0348] Optionally, the accelerator chip 1120 can perform various functions of the accelerator 1100 by running or executing software programs stored in the memory 1130 and by calling data stored in the memory 1130. In a specific implementation, as one embodiment, the accelerator chip 1120 may include one or more GPUs.

[0349] Optionally, the accelerator 1100 may also include multiple accelerator chips. Each of these accelerator chips may be a single-core accelerator chip (single-GPU) or a multi-core accelerator chip (multi-GPU). Here, an accelerator chip may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0350] The memory 1130 is used to store the software program for executing the model inference in this application, and its execution is controlled by the accelerator chip 1120. The specific implementation can be referred to the model inference in the above method embodiments, and will not be repeated here. For example, the memory 1130 can be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it can be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 1130 can be integrated with the accelerator chip 1120, or it can exist independently and be accessed through the interface circuit of the accelerator 1100 (…). Figure 11 (Not shown in the image) is coupled to the acceleration chip 1120, and this embodiment does not specifically limit this.

[0351] Transceiver 1110 is used for communication with other devices. For example, if accelerator 1100 is a user terminal (such as a client) or application server, transceiver 1110 can be used to communicate with the accelerator chip or with another processing chip. Alternatively, if accelerator 1100 is a multi-core chip, transceiver 1110 can be used to communicate with another multi-core chip.

[0352] Optionally, transceiver 1110 may include a receiver and a transmitter. Figure 11 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. Optionally, the transceiver 1110 can be integrated with the accelerator chip 1120, or it can exist independently and be connected through the interface circuit of the accelerator 1100 (…). Figure 11 (Not shown in the image) is coupled to the acceleration chip 1120, and this embodiment does not specifically limit this.

[0353] In this embodiment, transceiver 1110 is used to: acquire a first inference request. Acceleration chip 1120 processes the first inference request according to a first hot weight to obtain a first inference result, and obtains an output result based on the first inference result and a second inference result. Accelerator 1100 and a second accelerator cooperate to process the first inference request. The second accelerator processes the first inference request according to a second hot weight to obtain a second inference result. Therefore, transceiver 1110 is also used to: receive the second inference result sent by the second accelerator. Accelerator 1100 can be used to implement the function and corresponding beneficial effects of accelerator a141 in the aforementioned embodiment, and the second accelerator can be used to implement the function and corresponding beneficial effects of accelerator b142 in the aforementioned embodiment, which will not be elaborated here.

[0354] In one possible scenario, the structure of the second accelerator can be referenced to the description of accelerator 1100 above, and will not be repeated here.

[0355] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0356] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0357] It is worth noting that if the computer program product can be provided to users through a cloud service subscription model, users can choose different subscription tiers according to their needs. For example, the computer program product can also provide enterprise-level customized services with professional domain customization, interface personalization, and extended functions according to the needs of users or enterprises. Furthermore, the computer program product provided in this application that can implement the above-mentioned model training methods can also be provided to users as a value-added service, and this application does not limit this. Alternatively, the computer program product can also be embedded into eDataMate™ or other large language model (large model) toolchain systems, such as operator acceleration libraries, etc.

[0358] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model reasoning method, characterized in that, The method is applied to a model inference system, which includes a first accelerator and a second accelerator, on which model weights are deployed. The method includes: During the processing of inference requests, the model inference system copies the target weights from the first accelerator to the second accelerator. The activation frequency or number of activations of the target weights is greater than the activation frequency or number of activations of the cold weights in the model.

2. The method according to claim 1, characterized in that, The target weight is a hot weight.

3. The method according to claim 1, characterized in that, Also includes: During the process of processing inference requests, the model inference system statistically analyzes the activation frequency or number of activations of the model's weights, and identifies a portion of the weights on the first accelerator as the target weights based on the statistical results.

4. The method according to claim 1, characterized in that: The target weight is the third sub-cold weight among the first cold weights configured in the first accelerator, and the activation frequency of the third sub-cold weight is greater than the activation frequency of the other weights among the cold weights excluding the third sub-cold weight.

5. The method according to claim 1, characterized in that: The target weights are not included in the weights originally deployed in the second accelerator.

6. The method according to claim 1, characterized in that: The first accelerator or the second accelerator is a GPU, an NPU, or a TPU.

7. A model reasoning system, characterized in that, The model inference system includes a first accelerator and a second accelerator, on which the weights of the model are deployed. The model inference system is used to copy the target weights from the first accelerator to the second accelerator during the processing of inference requests, wherein the activation frequency or number of activations of the target weights is greater than the activation frequency or number of activations of the cold weights in the model.

8. The system according to claim 7, characterized in that, The target weight is a hot weight.

9. The system according to claim 7, characterized in that, The reasoning system is also used for: During the process of processing inference requests, the model inference system statistically analyzes the activation frequency or number of activations of the model's weights, and identifies a portion of the weights on the first accelerator as the target weights based on the statistical results.

10. The system according to claim 7, characterized in that, The target weight is the third sub-cold weight among the first cold weights configured in the first accelerator, and the activation frequency of the third sub-cold weight is greater than the activation frequency of the other weights among the cold weights excluding the third sub-cold weight.

11. The system according to claim 7, characterized in that, The target weights are not included in the weights originally deployed in the second accelerator.

12. The system according to claim 7, characterized in that: The first accelerator or the second accelerator is a GPU, an NPU, or a TPU.

13. A server, characterized in that, Includes processor, multiple accelerators, and memory. The one or more memories are used to store computer programs or instructions. The processor or accelerator is used to execute computer programs or instructions stored in the one or more memories to cause the server to perform the method as described in any one of claims 1 to 6.

14. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processing device, cause the processing device to perform the method described in any one of claims 1 to 6.

Images