Model inference method and system

Abstract

Disclosed are a model inference method and system, relating to the technical field of AI. The model inference system acquires a first inference request; a first accelerator processes the first inference request on the basis of a first hot weight to obtain a first inference result; a second accelerator processes the first inference request on the basis of a second hot weight to obtain a second inference result; and then the first accelerator obtains an output result on the basis of the first inference result and the second inference result, wherein the activation frequency of the first hot weight is greater than that of a cold weight in a model, and the first hot weight is identical to the second hot weight. The identical hot weights are deployed in the first accelerator and the second accelerator, and thus, when performing model inference, the first accelerator and the second accelerator can locally retrieve the hot weights to perform model inference. Since a speed at which the accelerators retrieve the hot weights locally is greater than a speed at which the accelerators retrieve the weights from other locations (e.g., a processor), the efficiency of model inference can be improved.

Description

Model inference method and system

[0001] The present application claims priority from the Chinese patent application No. 202411280640.2 filed on September 12, 2024, and entitled "Model inference method and system", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a model inference method and system. BACKGROUND

[0003] Model inference refers to a process of using a trained model to process input data to output a processing result. In the foregoing process, a computing device obtains weights of the model from a memory to process the input data. In a single-site scenario, a part of the weights is often deployed in an acceleration card, and another part of the weights is deployed in a processor. When the acceleration card processes the input data, the weights are obtained from the processor to process the input data. However, the communication between the acceleration card and the processor is often performed through a peripheral component interconnect express (PCIe) with limited bandwidth, and the acceleration card obtains the weights at a slow speed, resulting in low efficiency of model inference. SUMMARY

[0004] The present application provides a model inference method and system to solve the problem of low efficiency of model inference of the acceleration card due to limited bandwidth of the PCIe and slow speed of the acceleration card in obtaining the weights from the processor.

[0005] In a first aspect, the present application provides a model inference method. The model inference method can be applied to a computer system or a computing device in the computer system that implements the model inference method, such as the computer system being a model inference system and the computing device including an accelerator. In one possible example, the model inference system includes a plurality of accelerators, the plurality of accelerators including a first accelerator and a second accelerator, the first accelerator being configured with first hot weights of a model, and the second accelerator being configured with second hot weights of the model. The model inference method provided by the present application includes: the model inference system obtaining a first inference request. The first accelerator processes the first inference request according to the first hot weights to obtain a first inference result, and the second accelerator processes the first inference request according to the second hot weights to obtain a second inference result. The first accelerator obtains an output result according to the first inference result and the second inference result. The activation frequency of the first hot weights is greater than the activation frequency of cold weights in the model, and the first hot weights are the same as the second hot weights.

[0006] In the present application, the same hot weights are deployed in the first accelerator and the second accelerator, and then the first accelerator and the second accelerator can obtain the hot weights locally to perform model inference when performing model inference. Since the speed of the accelerator (the first accelerator and the second accelerator) obtaining the hot weights locally is greater than the speed of the accelerator obtaining the weights from other locations (such as a processor), the efficiency of model inference can be improved. In addition, since the activation frequency of the hot weights is greater than the activation frequency of the cold weights in the model, when the first accelerator and the second accelerator process the first inference request, most of the weights required to execute the first inference request can be obtained locally, that is, the amount of data of the first accelerator and the second accelerator obtaining the weights from other locations is reduced. In the case where the speed of the accelerator obtaining the hot weights locally is greater than the speed of the accelerator obtaining the weights from other locations, the efficiency of model inference can be further improved.

[0007] In one possible case, the above-mentioned model can be a large language model (LLM), a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), etc.

[0008] In one possible case, the weights corresponding to a network layer in the above-mentioned model include the first hot weights.

[0009] In another possible case, the weights corresponding to a plurality of network layers in the above-mentioned model include the first hot weights.

[0010] In a possible example, the model inference system obtains the first inference request, including: the first accelerator obtains the first inference request.

[0011] In another possible example, the model inference system further includes a processor, and the model inference system obtains the first inference request, including: the processor obtains the first inference request.

[0012] For example, the processor obtains the first inference request through the transceiver.

[0013] In a possible implementation, the first accelerator and the second accelerator are of the same type.

[0014] In a possible example, the first accelerator and the second accelerator are both processors for indicating model inference.

[0015] For example, the first accelerator and the second accelerator are both graphics processing units (GPUs).

[0016] For another example, the first accelerator and the second accelerator are both neural network processing units (NPUs).

[0017] For another example, the first accelerator and the second accelerator are both tensor processing units (TPUs).

[0018] For another example, the first accelerator and the second accelerator are both field-programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs), or data processing units (DPUs), and the like.

[0019] In a possible implementation, the plurality of accelerators further include a third accelerator, the third accelerator is of a type different from the first accelerator and the second accelerator, the third accelerator is configured with a third hot weight of the model, and the third hot weight is different from the first hot weight. Further, the model inference method further includes: the third accelerator processes the first inference request according to the third hot weight to obtain a third inference result. The first accelerator obtains the output result according to the first inference result and the second inference result, including: the first accelerator obtains the output result according to the first inference result, the second inference result, and the third inference result.

[0020] In the present application, since the model inference system includes different types of accelerators, and the different types of accelerators store different hot weights, the model inference can be realized through the heterogeneous model inference system, which can flexibly cope with different inference requirements and loads, thereby improving the model inference efficiency.

[0021] In one possible case, the types of the first accelerator and the second accelerator are GPUs, and the type of the third accelerator is an NPU.

[0022] In another possible case, the types of the first accelerator and the second accelerator are GPUs, and the type of the third accelerator is a TPU.

[0023] In one possible example, the first hot weight stored in the first accelerator is the weight of a first network layer in the model, the third hot weight stored in the third accelerator is the weight of a second network layer in the model, and the first network layer and the second network layer are different network layers in the model.

[0024] In one possible implementation, the first accelerator processes the first inference request according to the first hot weight to obtain a first inference result, including: the first accelerator processes the first inference request using a first sub-hot weight in the first hot weight to obtain the first inference result. The first sub-hot weight is a weight required by the first hot weight to process the first inference request, which is predicted by the flyline predictor of the model inference system.

[0025] In the present application, the first accelerator only activates the first sub-hot weight when processing the first inference request, that is, the first accelerator only acquires the first sub-hot weight, thereby reducing the data amount of the first accelerator to acquire the weight, and further reducing the time required to acquire the weight, thereby improving the efficiency of the first accelerator to perform model inference.

[0026] Similarly, in one possible implementation, the second accelerator processes the first inference request according to the second hot weight to obtain a second inference result, including: the second accelerator processes the first inference request using a second sub-hot weight in the second hot weight to obtain the second inference result. The second sub-hot weight is a weight required by the first hot weight to process the first inference request, which is predicted by the flyline predictor of the model inference system.

[0027] In the present application, the second accelerator only activates the second sub-hot weight when processing the first inference request, that is, the second accelerator only acquires the second sub-hot weight, thereby reducing the data amount of the second accelerator to acquire the weight, and further reducing the time required to acquire the weight, thereby improving the efficiency of the second accelerator to perform model inference.

[0028] In a possible implementation, the first accelerator is further configured with a first cold weight in the cold weights, and the second accelerator is further configured with a second cold weight in the cold weights. The first accelerator processes the first inference request according to the first hot weight to obtain the first inference result, including that the first accelerator processes the first inference request according to the first hot weight and the first cold weight to obtain the first inference result. The second accelerator processes the first inference request according to the second hot weight to obtain the second inference result, including that the second accelerator processes the first inference request according to the second hot weight and the second cold weight to obtain the second inference result.

[0029] In the present application, since the first accelerator is configured with the first cold weight and the second accelerator is configured with the second cold weight, the amount of data that the first accelerator or the second accelerator acquires from other devices in the model inference system can be reduced, and the time required for acquiring the weights is reduced, thereby improving the efficiency of the model inference of the model inference system.

[0030] In a possible case, the distance between the storage area of the first hot weight and the acceleration chip in the first accelerator is less than or equal to the distance between the storage area of the first cold weight and the acceleration chip, and the access efficiency of the acceleration chip to the first hot weight is greater than or equal to the access efficiency of the acceleration chip to the first cold weight.

[0031] In a possible example, the first hot weight is stored in a first-level cache, a second-level cache, or a third-level cache in the first accelerator, and the first cold weight is stored in a first-level cache, a second-level cache, or a third-level cache in the second accelerator. In a possible implementation, the model inference system predicts the weights required to be used when processing the first inference request by the model inference system through a flying wire predictor. The required weights include a first sub-hot weight in the first hot weight, a first sub-cold weight in the first cold weight, a second sub-hot weight in the second hot weight, and a second sub-cold weight in the second cold weight. The first accelerator processes the first inference request according to the first hot weight and the first cold weight to obtain the first inference result, including that the first accelerator processes the first inference request according to the first sub-hot weight and the first sub-cold weight to obtain the first inference result. The second accelerator processes the first inference request according to the second hot weight and the second cold weight to obtain the second inference result, including that the second accelerator processes the first inference request according to the second sub-hot weight and the second sub-cold weight to obtain the second inference result.

[0032] In the present application, the first accelerator only activates the first sub-hot weight and the first sub-cold weight when processing the first inference request, that is, the first accelerator only acquires the first sub-hot weight and the first sub-cold weight, thereby reducing the data amount of the first accelerator acquiring the weights (from the data amount of the first hot weight and the first sub-cold weight to the data amount of the first sub-hot weight and the first sub-cold weight), and further reducing the time required for the first accelerator to acquire the weights, thereby improving the efficiency of model inference.

[0033] Similarly, the second accelerator only activates the second sub-hot weight and the second sub-cold weight when processing the first inference request, that is, the second accelerator only acquires the second sub-hot weight and the second sub-cold weight, thereby reducing the data amount of the second accelerator acquiring the weights (from the data amount of the second hot weight and the second sub-cold weight to the data amount of the second sub-hot weight and the second sub-cold weight), and further reducing the time required for the second accelerator to acquire the weights, thereby improving the efficiency of model inference. In summary, the efficiency of model inference of the model inference system can be improved.

[0034] In a possible implementation, the difference between the sum of the data amount of the first sub-hot weight and the data amount of the first sub-cold weight and the sum of the data amount of the second sub-hot weight and the data amount of the second sub-cold weight is less than or equal to a first threshold.

[0035] In the present application, since the difference between the sum of the data amount of the first sub-hot weight and the data amount of the first sub-cold weight and the sum of the data amount of the second sub-hot weight and the data amount of the second sub-cold weight is less than or equal to the first threshold, the difference in the required hardware resources (computing resources, storage resources, network resources) of each accelerator (the first accelerator and the second accelerator) when processing the first inference request is within the first threshold, in other words, each accelerator is load balanced when processing the first inference request, thereby improving the efficiency of model inference of the model inference system.

[0036] In a possible implementation, the first cold weight includes a weight combination with a co-occurrence greater than or equal to a second threshold, and the weight combination is stored in the same storage unit in the first accelerator.

[0037] Similarly, the second cold weight includes a weight combination with a co-occurrence greater than or equal to a second threshold, and the weight combination is stored in the same storage unit in the second accelerator.

[0038] In the present application, the first accelerator stores the weight combination in the same storage unit, and the first accelerator can acquire the weight combination from a smaller storage area, thereby speeding up the acquisition of the weight and improving the efficiency of model inference.

[0039] In a possible case, the above-mentioned weight combination can include at least two weights in the weight matrix corresponding to the model.

[0040] In a possible scenario, the storage unit is a block, a cache line, a segment, or a physical storage chunk.

[0041] In a possible implementation, the model inference method further includes: the first accelerator obtaining a first weight switching request indicating switching of a first cold weight configured in the first accelerator, and switching a third sub-cold weight in the first cold weight to a hot weight according to the first weight switching request. The activation frequency of the third sub-cold weight is greater than the activation frequency of other weights in the cold weight except the third sub-cold weight.

[0042] In the present application, the first accelerator switches the third sub-cold weight in the first cold weight to the hot weight, reduces the amount of data that the first accelerator obtains from the remote first cold weight in the case that the storage area of the first cold weight is farther than the storage area of the first hot weight, thereby reducing the time consumption of the first accelerator in obtaining the weight, and improving the efficiency of model inference.

[0043] In a possible implementation, the model inference method further includes: the first accelerator obtaining a second weight switching request indicating switching of a first hot weight configured in the first accelerator, and switching a third sub-hot weight in the first hot weight to a cold weight according to the second weight switching request. The activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weight except the third sub-hot weight.

[0044] In the present application, the first accelerator switches the third sub-hot weight in the first hot weight to the cold weight, reduces the amount of data of the first hot weight stored in the first accelerator, reduces the resource load of the first accelerator, and reduces the power consumption of the first accelerator.

[0045] In a possible implementation, the model inference method further includes: the first accelerator obtaining a third weight switching request indicating switching of a weight configured in the first accelerator, and switching the third sub-hot weight in the first hot weight to the cold weight and switching a third sub-cold weight in the first cold weight configured in the first accelerator to the hot weight according to the third weight switching request. The activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weight except the third sub-hot weight, and the activation frequency of the third sub-cold weight is greater than the activation frequency of other weights in the cold weight except the third sub-cold weight.

[0046] In the present application, the first accelerator reduces the amount of data of the weight from the first cold weight by switching the third sub-hot weight in the first hot weight to a cold weight and switching the third sub-cold weight in the first cold weight to a hot weight, thereby reducing the time consumption of the first accelerator in acquiring the weight, and improving the efficiency of model inference.

[0047] In a possible implementation, the model inference system further includes a processor configured to manage the plurality of accelerators. The processor acquires the first inference request, and determines the weights required by the first accelerator and the second accelerator to process the first inference request based on the fly-line prediction of the first inference request, and sends the identifiers of the weights required by the first accelerator and the second accelerator to process the first inference request to the first accelerator and the second accelerator respectively.

[0048] In the present application, the processor sends the identifiers of the weights required by the first accelerator and the second accelerator to process the first inference request to the first accelerator, so that the first accelerator and the second accelerator selectively activate part of the weights in the model, thereby reducing the number of weights acquired by the first accelerator and the second accelerator when processing the first inference request, reducing the time consumption required for acquiring the weights, and improving the efficiency of model inference of the model inference system.

[0049] In a possible case, the identifier is a storage address of the required weight.

[0050] In a possible implementation, the processor is further configured with a count matrix configured to indicate the activation frequency of the weights in the model. The model inference method further includes: determining, by the processor, a first position of the required weight in the weight matrix based on the weights required by the model inference system to process the first inference request, and increasing the value at a second position corresponding to the first position in the count matrix by 1. If the value at a third position in the count matrix is greater than a third threshold value, the first accelerator is sent a first weight switching request; if the value at a fourth position in the count matrix is less than a fourth threshold value, the first accelerator is sent a second weight switching request; if the value at the third position in the count matrix is greater than the third threshold value and the value at the fourth position in the count matrix is less than the fourth threshold value, the first accelerator is sent a third weight switching request.

[0051] In the present application, the processor counts the activation times of the required utilized weights during processing the inference request, and then sends a weight switching request (first weight switching request, second weight switching request, third weight switching request) to the first accelerator when the threshold (third threshold, fourth threshold) requirement is met. The processor dynamically switches the cold weights and hot weights according to the real inference request, realizes that the first hot weights are stored in the first accelerator, reduces the data amount of the first accelerator for acquiring the weights from other devices included in the model inference system when processing the inference request, thereby reducing the time consumption of the first accelerator for acquiring the weights, and improving the inference efficiency of the model.

[0052] In a possible case, the weight corresponding to the position corresponding to the third position in the weight matrix is configured in the first accelerator, and the weight corresponding to the position corresponding to the fourth position in the weight matrix is configured in the first accelerator.

[0053] In a possible case, the weight corresponding to the position corresponding to the third position in the weight matrix is configured in the first accelerator, and the weight corresponding to the position corresponding to the fourth position in the weight matrix is configured in the first accelerator.

[0054] The model inference system is configured to obtain a first inference request.

[0055] The first accelerator is configured to process the first inference request according to the first hot weight, and obtain a first inference result; and the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0056] The second accelerator is configured to process the first inference request according to the second hot weight, and obtain a second inference result; and the second hot weight is the same as the first hot weight.

[0057] The first accelerator is configured to obtain an output result according to the first inference result and the second inference result.

[0058] In a possible implementation, the first accelerator and the second accelerator are of the same type.

[0059] In a possible implementation, the plurality of accelerators further include a third accelerator, the third accelerator being different from the first accelerator and the second accelerator in type. The third accelerator is configured with a third hot weight of the model, the third hot weight being different from the first hot weight. The third accelerator is configured to process the first inference request according to the third hot weight to obtain a third inference result. The first accelerator is specifically configured to obtain the output result according to the first inference result, the second inference result, and the third inference result.

[0060] In a possible implementation, the first accelerator is further configured with a first cold weight of the cold weights, and the second accelerator is further configured with a second cold weight of the cold weights. The first accelerator is specifically configured to process the first inference request according to the first hot weight and the first cold weight to obtain the first inference result. The second accelerator is specifically configured to process the first inference request according to the second hot weight and the second cold weight to obtain the second inference result.

[0061] In a possible implementation, the first accelerator is specifically further configured to process the first inference request according to a first sub-hot weight of the first hot weight and a first sub-cold weight of the first cold weight to obtain the first inference result. The second accelerator is specifically further configured to process the first inference request according to a second sub-hot weight of the second hot weight and a second sub-cold weight of the second cold weight to obtain the second inference result.

[0062] In a possible implementation, a difference between a sum of a data amount of the first sub-hot weight and a data amount of the first sub-cold weight and a sum of a data amount of the second sub-hot weight and a data amount of the second sub-cold weight is less than or equal to a first threshold value.

[0063] In a possible implementation, the first cold weight includes a weight combination whose co-occurrence is greater than or equal to a second threshold value, and the weight combination is stored in a same storage unit in the first accelerator.

[0064] In a possible implementation, the first accelerator is further configured to obtain a first weight switching request, the first weight switching request being used to indicate switching of the first cold weight configured in the first accelerator. The first accelerator is further configured to switch, according to the first weight switching request, a third sub-cold weight of the first cold weight to a hot weight, an activation frequency of the third sub-cold weight being greater than an activation frequency of other weights in the cold weights except the third sub-cold weight.

[0065] In a possible implementation, the first accelerator is further configured to obtain a second weight switching request, the second weight switching request being used to indicate switching of a first hot weight configured in the first accelerator. The first accelerator is further configured to switch a third sub-hot weight in the first hot weight to a cold weight according to the second weight switching request, an activation frequency of the third sub-hot weight being less than an activation frequency of other weights in the first hot weight except the third sub-hot weight.

[0066] In a possible implementation, the first accelerator is further configured to obtain a third weight switching request, the third weight switching request being used to indicate switching of a weight configured in the first accelerator. The first accelerator is further configured to switch a third sub-hot weight in the first hot weight to a cold weight and switch a third sub-cold weight in the first cold weight configured in the first accelerator to a hot weight according to the third weight switching request. An activation frequency of the third sub-hot weight is less than an activation frequency of other weights in the first hot weight except the third sub-hot weight, and an activation frequency of the third sub-cold weight is greater than an activation frequency of other weights in the cold weight except the third sub-cold weight.

[0067] For more detailed implementation of the model inference system, refer to the description of the first aspect or any implementation of the first aspect, and the content of the following specific embodiments, which will not be repeated here.

[0068] In a third aspect, the present application provides a chip, comprising: a control circuit and an interface circuit. The interface circuit is configured to obtain a first inference request, and the control circuit is configured to execute the processing of the first request by the first accelerator or the second accelerator in the method of the first aspect or any possible implementation of the first aspect. The chip provided in the present application can be a chip included in the first accelerator or the second accelerator in the first aspect for performing calculation in the model inference process.

[0069] In a fourth aspect, the present application provides a server. The server comprises a processor, a plurality of accelerators and a memory.

[0070] The processor is configured to execute instructions stored in the memory to obtain a first inference request.

[0071] A first accelerator in the plurality of accelerators is configured with a first hot weight of a model, and the first accelerator processes the first inference request according to the first hot weight to obtain a first inference result; an activation frequency of the first hot weight is greater than an activation frequency of a cold weight in the model.

[0072] A second accelerator in the plurality of accelerators is configured with a second hot weight of the model, and the second accelerator processes the first inference request according to the second hot weight to obtain a second inference result; the second hot weight is the same as the first hot weight.

[0073] The first accelerator obtains an output result according to the first inference result and the second inference result.

[0074] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster comprises at least one computing device, each computing device comprising a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to execute the computer instructions to implement the method in the first aspect or any possible implementation manner of the first aspect. The computing device cluster provided in the present application can be the model inference system in the second aspect, i.e., the computing device cluster comprises the first accelerator and the second accelerator.

[0075] In a sixth aspect, the present application provides a computer readable storage medium, which stores computer programs or instructions. When the computer programs or instructions are executed by a processing device, the method in the first aspect or any possible implementation manner of the first aspect is implemented.

[0076] In a seventh aspect, the present application provides a computer program product, which comprises computer programs or instructions. When the computer programs or instructions are executed by a processing device, the method in the first aspect or any possible implementation manner of the first aspect is implemented.

[0077] The beneficial effects of the second aspect to the seventh aspect can refer to the first aspect or any possible implementation manner of the first aspect, which will not be repeated here. On the basis of the implementation manners provided in the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0078] FIG. 1 is a structural schematic diagram of a data processing system provided by the present application;

[0079] FIG. 2 is a schematic diagram of model distributed inference provided by the present application;

[0080] FIG. 3 is a flow schematic diagram of a model inference method provided by the present application;

[0081] FIG. 4 is a schematic diagram of cold-hot weight determination provided by the present application;

[0082] FIG. 5 is a schematic diagram of training of an activation classifier provided by the present application;

[0083] FIG. 6 is a schematic diagram of cold-hot weight configuration provided by the present application;

[0084] FIG. 7 is a schematic diagram of cold-hot weight configuration provided by the present application;

[0085] FIG. 8 is a schematic diagram of load balancing distribution provided by the present application;

[0086] FIG. 9 is a flowchart of a cold-hot weight switching method provided by the present application;

[0087] FIG. 10 is a flowchart of a model inference method provided by the present application;

[0088] FIG. 11 is a structural diagram of an accelerator provided by the present application. DETAILED DESCRIPTION

[0089] First, the technical terms involved in the present application are introduced.

[0090] A large language model (LLM) can effectively process massive data and provide efficient and accurate information processing and analysis services for users by virtue of its powerful computing power and complex algorithms. LLM not only performs outstandingly in understanding and generating human language, but also exhibits great potential in solving complex problems and tasks. For example, LLM is widely used in automated question and answer systems, text summary generation, machine translation, and language generation, greatly improving efficiency and accuracy. In particular, when dealing with large-scale data sets, LLM can mine deep patterns and correlations from them to support decision-making. In addition, the self-learning ability of LLM enables it to continuously evolve, and the performance and intelligence of LLM are constantly improving through continuous learning of new data. Common LLMs include the Bidirectional Encoder Representations from Transformers (BERT) model based on the transformer, which can be pre-trained using unlabelled text to train deep bidirectional representations. In some optional cases, LLM can also be referred to as a large model. The large model provided by the embodiments of the present application can not only refer to LLM, but also refer to a model whose model parameters reach a certain degree. Depending on the field in which the large model is applied, the large model can also refer to a model that contains various functions such as image processing, human-computer interaction, semantic search, semantic query, or dialogue, etc. The present application does not limit the field to which the large model can be applied or the specific name. In this text, for the sake of simplicity, the large model is named, but this should not be understood as a limitation of the present application, and the following will not be described in detail.

[0091] Tensor parallelism (TP) refers to splitting the weights of each network layer of the model into multiple parts, and configuring one part of the weights of each network layer of the model into one accelerator. The model is deployed in multiple accelerators in a tensor parallel manner, and each accelerator processes the input data to obtain multiple processing results. The multiple processing results are aggregated to obtain the output result corresponding to the input data.

[0092] Data parallelism (DP) refers to splitting data into multiple batches or multiple parts, and then using multiple accelerators deployed with the same model to process the data in multiple batches or multiple parts respectively, with one accelerator processing the data in one batch or one part. Alternatively, multiple accelerators deployed with the same network layer in the model are used to process the data in multiple batches or multiple parts respectively, with one accelerator processing the data in one batch or one part.

[0093] Pipeline parallelism (PP) refers to configuring each network layer in the model to a different accelerator. After input data is input to an accelerator deployed with the first network layer in the model, the processing result of the input data by the accelerator is input to an accelerator deployed with the second network layer in the model, and so on. After processing by an accelerator deployed with the last network layer in the model, an output result is obtained.

[0094] The current LLM has a parameter (also referred to as weight) of the order of 100 billion based on a transformer structure. In the process of using the LLM to perform model inference, according to input data such as a query statement input by a tenant, the LLM can generate a corresponding output result such as an answer to the query statement through a self-recurrent manner.

[0095] Due to the large weight size of the LLM, the single card (accelerator) memory wall is affected, and therefore, the LLM is often deployed in multiple cards such as a multi-machine multi-card inference server or a supernode in a TP, DP, PP, etc. manner. Since the weight matrix in the LLM is usually a thin and long matrix, the accelerator will show the characteristics of low computation and large memory access when using the thin and long matrix for model inference, resulting in a long time for the accelerator to obtain the weight matrix and affecting the model inference efficiency.

[0096] To solve the above problems, two possible processing schemes are given below.

[0097] Processing scheme 1 reduces the model weight by quantizing and / or pruning the LLM, thereby reducing the data amount of the accelerator to obtain the weight matrix, and further reducing the time for the accelerator to obtain the weight matrix and improving the model inference efficiency. However, the LLM and the traditional model (such as a convolutional neural network, a recurrent neural network, etc.) have great differences in network structure, and directly reusing the traditional quantization / pruning manner will cause great loss of inference accuracy of the LLM.

[0098] The processing scheme 2 deploys one model into more accelerators by using a TP, a DP, a PP, etc., reduces the model weight configured in a single accelerator, but more accelerators will cause the weight of the model to be more discrete, and thus will bring greater communication cost (the amount of data communicated between accelerators increases), causing the speed of the accelerator to obtain the weight to be slow, and thus the efficiency of the multiple accelerator cards to perform model inference changes little.

[0099] Based on this, the present application provides a model inference method. The model inference method is applied to a model inference system including a first accelerator and a second accelerator, the first accelerator is configured with a first hot weight of a model, and the second accelerator is configured with a second hot weight of the model. The model inference system obtains a first inference request. The first accelerator processes the first inference request according to the first hot weight to obtain a first inference result, and the second accelerator processes the first inference request according to the second hot weight to obtain a second inference result, and then the first accelerator obtains an output result according to the first inference result and the second inference result. Wherein, the first hot weight and the second hot weight are the same, the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model, and the first hot weight and the second hot weight are the same.

[0100] In the present application, the same hot weight is deployed in the first accelerator and the second accelerator, and then the first accelerator and the second accelerator can obtain the hot weight from the local to perform model inference when performing model inference. Since the speed of the accelerator (the first accelerator and the second accelerator) to obtain the hot weight from the local is greater than the speed of the accelerator to obtain the weight from other positions (such as a processor), the efficiency of the model inference can be improved. In addition, since the activation frequency of the hot weight is greater than the activation frequency of the cold weight in the model, when the first accelerator and the second accelerator process the first inference request, most of the weights required to execute the first inference request can be obtained from the local, that is, the amount of data of the first accelerator and the second accelerator to obtain the weight from other positions is reduced. In the case that the speed of the accelerator to obtain the hot weight from the local is greater than the speed of the accelerator to obtain the weight from other positions, the efficiency of the model inference can be further improved.

[0101] Next, the model inference method provided by the present application will be described in detail in combination with the drawings.

[0102] Fig. 1 is a structural schematic diagram of a data processing system provided by the present application. As shown in Fig. 1, the data processing system includes a computing device 110, a model training system 120, a database 130, a model inference system 140, a terminal device 150, a data storage system 160, and a data acquisition device 170.

[0103] The computing device 110 can be configured to interact with a user, e.g., the computing device 110 can obtain input data input by the user via interaction with a terminal device 150 on the user side. In one possible scenario, the computing device 110 can be further configured to pre-process the input to obtain pre-processed data, and send the pre-processed data to the model inference system 140.

[0104] In one possible scenario, the computing device 110 can further be configured to perform embedded expression on the pre-processed data to obtain a corresponding vector.

[0105] When the input data is a query statement, the computing device 110 can perform data cleaning, denoising, etc. on the query statement to obtain a pre-processed query statement, and perform embedded expression on the pre-processed query statement to obtain a corresponding vector.

[0106] In another possible scenario, the computing device 110 can be further configured to pre-process training data sent by the terminal device 150, and store the pre-processed training data to the database 130.

[0107] In one possible scenario, the data storage system 160 can be configured to temporarily store the pre-processed data, the vector, or the output result output by the model inference system 140. Then, the computing device 110 can take the input data and the corresponding output result as input data again, and store the input data to the database 130.

[0108] The terminal device 150 can be configured to receive operations of a user, e.g., the terminal device 150 can receive input data or training data input by the user.

[0109] The training data stored in the database 130 can not only come from the pre-processed training data of the computing device 110, but also include training data collected by a data collection device 170. The data collection device 170 can be configured to collect training data, and store the training data to the database 130.

[0110] The model training system 120 can be configured to train the neural network 101. For example, the model training system 120 can be configured to train the neural network using the training data in the database 130 until the loss function in the neural network converges, and the loss function value is less than a certain threshold, and the training of the neural network is completed, so that the neural network 101 reaches a certain accuracy. In one possible example, the model training system 120 can deploy the trained neural network 101 to the model inference system 140. The neural network 101 can be a large model as described above.

[0111] The neural network 101 is deployed in the model inference system 140, and the model inference system 140 performs model inference according to input data input by a user to obtain an output result. In other words, the model inference system 140 inputs the input data into the neural network 101, and outputs the output result.

[0112] The model inference system 140 can return the output result to the terminal device 150, so as to provide the output result to the user for viewing.

[0113] Taking the input data as a query statement as an example, the model inference system 140 inputs the query statement into the neural network 101 to obtain a query result corresponding to the query statement, and returns the query result to the terminal device 150.

[0114] In a possible case, the model inference system 140 can receive the input data sent by the terminal device 150.

[0115] In another possible case, the model inference system 140 can receive the preprocessed input data sent by the computing device 110.

[0116] It should be noted that in actual application, the training data maintained in the database 130 does not necessarily come from the data acquisition device 170, but can also be received from other devices. In addition, the model training system 120 does not necessarily train the neural network 101 based on the training data maintained in the database 130, but can also obtain the training data from the cloud or other places to train the neural network 101. The above description should not be regarded as a limitation of the embodiments of the present application.

[0117] In the present embodiment, the neural network 101 can be referred to as an AI model or a model, which can be an LLM or other models.

[0118] In an optional example, the neural network 101 refers to an LLM. Since the LLM contains a large number of processing layers, it takes a long time to train / infer the LLM by using a single device. Therefore, the distributed training method is used to train / infer the LLM. For details, refer to the embodiments provided in FIGS. 2 to 10 below, which are not described herein.

[0119] In another alternative example, the neural network 101 refers to other types of networks. For example, the neural network 101 is a convolutional neural network (CNN), a recurrent neural network (RNN), or a graph neural network (GNN), etc. More implementations of the CNN, RNN, or GNN can refer to the description of the general technology, which will not be described herein.

[0120] The functions of the computing device 110, the model training system 120, the database 130, the model inference system 140, the terminal device 150, the data storage system 160, and the data collection device 170 are introduced above. The possible structures of the computing device 110, the model training system 120, the database 130, the model inference system 140, the terminal device 150, the data storage system 160, and the data collection device 170 are described below.

[0121] The computing device 110 can be a terminal, such as a computer, a mobile phone terminal, a tablet computer, a notebook computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera, or a vehicle-mounted computer, etc. The computing device 110 can also be an edge device (for example, a box with a processing-capable chip) or the like. In this application, the computing device 110 can be a computing device connected to a base station or a computing device deployed in a base station, such as a server or a cloud device, etc.

[0122] The model training system 120 can include one or more processing devices, which can be terminals or other computing devices supporting integer or floating point calculations, such as servers, cloud devices, or chip systems, etc.

[0123] In a possible embodiment, the computing device 110 and the processing device included in the model training system 120 are different physical devices. For example, the computing device 110 and the processing device included in the model training system 120 are chip systems on different physical devices. For example, the computing device 110 can be a GPU, a central processor, another general-purpose processor, a digital signal processing (DSP) chip, an ASIC, an FPGA, or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0124] In another possible embodiment, the computing device 110 and the processing device included in the model training system 120 are deployed on the same physical device, or the computing device 110 and the processing device included in the model training system 120 are the same physical device.

[0125] The data collection device 170 and the processing device included in the computing device 110 and the model training system 120 can be the same or different devices. In an embodiment of the present application, the data collection device 170 can be a video camera, a camera, a mobile phone, a tablet computer, or a computing device or network device with data collection functions.

[0126] In some embodiments, if the computing device 110 and the processing device included in the model training system 120 are the same physical device, the model training system 120 can configure the trained neural network 101 to itself, and use the trained neural network 101 to achieve the target function of the model, such as identification or document verification in a business district, school, park, or sports venue in a city, or such as target detection, object recognition, or classification operations on data, or such as face payment, object classification (such as commodity classification), and the like.

[0127] In other embodiments, the model training system 120 can configure the trained neural network 101 to the model inference system 140, which includes multiple accelerators, so that the model inference system can achieve the target function of the model as described above. In addition to the target functions of the above embodiments, the neural network 101 can also achieve some functions that can be achieved by LLM, and the like.

[0128] Further, according to the functions performed by the computing device 110, the computing device 110 can be further subdivided into the architecture as shown in FIG. 1. As shown in FIG. 1, the computing device 110 is configured with an I / O interface 111 and a preprocessing module 112.

[0129] The I / O interface 111 is used for data interaction with external devices. A user can input data to the I / O interface 111 through a terminal device 150, such as an instruction for instructing the computing device 110 to start feature extraction or model training on training data. In addition, the input data can also come from a database 130.

[0130] The preprocessing module 112 is used for preprocessing according to the input data received by the I / O interface 111. In an embodiment of the present application, the preprocessing module 112 can be used to generate training data, such as a training set, a validation set, and a test set, according to the input data received from the I / O interface 111. Optionally, the preprocessing module 112 can also perform preprocessing operations such as denoising on input data such as feature vectors corresponding to query statements, and the like, to eliminate irrelevant information and restore useful real information.

[0131] In the process of pre-processing the input data by the computing device 110, the computing device 110 can call data, code, etc. in the data storage system 160 for corresponding processing, and can also store the data and instructions obtained by the corresponding processing in the data storage system 160.

[0132] Finally, the I / O interface 111 sends the processing result to the model inference system 140 for further processing of the aforementioned processing result by the model deployed in the model inference system 140, to obtain an output result.

[0133] The terminal device 150 can be a data collection end, which collects the input data of the input I / O interface 111 and the output result of the output I / O interface 111 as new sample data as shown in FIG. 1, and stores the sample data in the database 130. Of course, the sample data can also be collected without the terminal device 150, and the input data of the input I / O interface 111 and the output result of the output I / O interface 111 as shown in FIG. 1 are stored in the database 130 as new sample data by the I / O interface 111.

[0134] The processing device included in the model inference system 140 can be a GPU, a NPU, a TPU, a central processing unit (CPU), other general-purpose processors, a DSP, an ASIC, an FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0135] In a possible case, the model inference system 140 includes a processor 140a and a plurality of accelerators, such as an accelerator a 141, an accelerator b 142, an accelerator c 143, and an accelerator d 144, etc.

[0136] In the model inference system 140 shown in FIG. 1, the processor 140a is configured to divide the model into a plurality of modules, each module including part of each network layer in the model, and further deploy the plurality of modules to the plurality of accelerators, one module being deployed in one accelerator, so as to improve the inference efficiency of the model. The processor 140a can be a CPU configured to receive a query statement and send a feature vector corresponding to the query statement to different accelerators, etc. In hardware implementation, the processor 140a can include one or more processing chips, which are not limited in the present application.

[0137] The accelerator provided in the embodiments of the present application includes one or more dies, such as the accelerator a 141 including die 1 and die 2, the accelerator b 142 including die 3 and die 4, the accelerator c 143 including die 5 and die 6, and the accelerator d 144 including die 7 and die 8. Taking the accelerator a 141 in the accelerator a 141, the accelerator b 142, the accelerator c 143, and the accelerator d 144 as an example, the accelerator a 141 can be, but is not limited to, a GPU chip, an NPU chip, a TPU chip, a micro processing chip, an ASIC chip, or one or more integrated circuit chips for controlling the execution of the computer program product provided in the present application, etc. Each accelerator includes a die, such as a GPU in a GPU chip, an NPU in an NPU chip, or a TPU in a TPU chip, etc. For example, if the die is a GPU in a GPU chip, the die can be used to perform mathematical and geometric calculations to achieve image rendering and other work.

[0138] In some possible implementations, different accelerators are networked in a full mesh mode (Full Mesh), that is, each accelerator is directly connected to each other. In some cases, the direct connection between different accelerators can be an inter-chip communication link, that is, chip to chip, such as connecting different accelerators in one or more of the following ways: a high-speed custom communication system (HCCS) interface, a high-speed interconnection bandwidth interface, an integrated circuit bus, a controller area network bus, a serial peripheral interface, a queue serial peripheral interface, a full-duplex asynchronous serial interface, or a half-duplex differential serial interface, etc. cache coherence system, HCCS) interface, a high-speed interconnection bandwidth interface, an integrated circuit bus, a controller area network bus, a serial peripheral interface, a queue serial peripheral interface, a full-duplex asynchronous serial interface, or a half-duplex differential serial interface, etc.

[0139] The HCCS interface is a high-speed connection channel between dies, which is used to promote data and calculation acceleration to obtain executable results. For example, in an accelerator, different dies are connected to each other using HCCS technology.

[0140] The high-speed interconnection bandwidth is a high-speed interconnection technology between GPUs, which is usually implemented by a plurality of pairs of wires printed on a computer board, and the two ends of a pair of wires are connected to different GPUs.

[0141] An inter-integrated circuit (I2C) bus is a source synchronous serial bus for short distance communication between different integrated circuits. I2C uses two lines for data transfer: a serial data line (SDL) and a serial clock line (SCL). A controller area network (CAN) bus is a serial communication protocol bus for real-time applications, which can use twisted pair wires to transmit signals. A serial peripheral interface (SPI) bus is a 3-wire synchronous serial full-duplex communication interface, which has the advantages of simple circuit, high speed, reliable communication, etc. A queued serial peripheral interface (QSPI) bus is based on SPI and adds a queue transmission mechanism. QSPI uses a dedicated communication interface to connect single, double, or four data lines. A full-duplex asynchronous serial interface is also called a universal asynchronous receiver / transmitter (UART) interface. The UART interface is a general-purpose serial data bus for asynchronous communication. The UART bus can be a bidirectional communication bus, and the UART interface converts the data to be transmitted between serial communication and parallel communication. For example, the UART interface refers to the RS-232 interface. A half-duplex differential serial interface is a serial communication bus interface. The half-duplex differential serial interface uses two-wire system, differential transmission and half-duplex mode, such as the RS-485 interface.

[0142] It is worth noting that, unlike the chip to chip communication link, in an accelerator, the dies can be connected by a die to die communication link, which can use but is not limited to an SIO interface.

[0143] In the embodiments of the present application, the transmission bandwidth (SIO bandwidth) of the die to die communication link between the dies is greater than the transmission bandwidth of the chip to chip communication link. The transmission bandwidth refers to the maximum amount of data that can be transmitted by different devices in a unit of time. For example, the dies in the same accelerator are connected by an SIO interface, and the transmission bandwidth (SIO bandwidth) of the communication link (or channel) connected by the SIO is greater than the transmission bandwidth of the communication link between different chips.

[0144] Taking the accelerator a 141 and the accelerator b 142 shown in FIG. 1 as an example, the SIO bandwidth between the core 1 and the core 2 is greater than the transmission bandwidth between the accelerator a 141 and the accelerator b 142. For example, the SIO bandwidth between the core 1 and the core 2 is 392 gigabytes per second (GB / s), and the transmission bandwidth between the accelerator a 141 and the accelerator b 142 is 100 GB / s.

[0145] FIG. 1 is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, elements, modules and the like shown in FIG. 1 does not constitute any limitation. According to the demand of a user for model inference, the model inference system 140 and the computing device 110 can include more or fewer hardware elements, which is not limited in the present application. For example, in FIG. 1, the data storage system 160 is an external memory relative to the computing device 110, and in other cases, the data storage system 160 can also be placed in the computing device 110.

[0146] Based on the model inference system 140 shown in FIG. 1, it is assumed that the model inference system 140 of an embodiment of the present application is deployed with an AI model based on a transformer architecture, and a possible example is shown in FIG. 2, which is a schematic diagram of model distributed inference provided by the present application. Please refer to FIG. 2, the model includes a plurality of transformer modules, such as transformer module 1, transformer module 2, and transformer module n, and n is a positive integer; the plurality of transformer modules are networked in a full mesh mode.

[0147] Different transformer modules rely on the model inference system 140 shown in FIG. 1 to be implemented, and each transformer module is configured into a different accelerator, and a plurality of accelerators are used to cooperatively process input data to improve the inference efficiency of the model. For example, different transformer modules in the model are respectively configured into different accelerators for inference. For example, the processor 140a configures the transformer module 1 into the accelerator a 141, configures the transformer module 2 into the accelerator b 142, and configures the transformer module n into the accelerator d 144. Other transformer modules can also be configured into other accelerators in the model inference system 140, which is not described here.

[0148] In the inference process of the model, the processor 140a acquires a sequence and inputs the sequence into the model, different accelerators are used to perform inference on the sequence by using transformer modules, and output the inference result corresponding to the sequence. It can be understood that different transformer modules in the model are inferred by different accelerators, realizing the parallel inference process of the model.

[0149] For example, the sequence is a vector corresponding to a query statement, and then the processor 140a sends the vector to different accelerators for processing, realizes that each accelerator uses a transformer module to infer the vector to obtain a corresponding inference result, and then the output result such as a query result is obtained after the inference results corresponding to each accelerator are summarized.

[0150] Among them, the sequence is an ordered data set, and each element in the sequence has a unique position. In some optional examples, the sequence is variable, and part or all of the elements can be modified; and the sequence can traverse the elements in it through iteration, and the device can also obtain the sub-sequence in the sequence through slicing. During the use of the sequence, one or more operations can also be performed on the sequence as needed, such as adding elements, deleting elements, sorting, searching, etc. For example, the elements in the sequence can include elements of any type, such as numbers, strings, objects, arrays, tuples, linked lists (or lists), tokens, stacks, and queues, etc. A string is a sequence composed of characters, commonly used in text processing, cryptography, image processing, etc. An array is a sequence composed of elements of the same type, commonly used to store a large amount of data and perform numerical calculations. A linked list is a sequence composed of nodes, commonly used to implement dynamic data structures and efficient insertion and deletion operations. A tuple is an immutable sequence, commonly used to package multiple values into a whole. A list is a variable sequence, commonly used to store and manipulate data. Stacks and queues are special sequences commonly used to implement data structures and algorithms.

[0151] Please refer to FIG. 2. Taking the vector of the query statement as an example, the query statement is semantic information generated by the processor 140a according to a sequence of data such as a piece of text or audio information, etc. The text or audio information is input by the user, transmitted by other devices, etc. The query statement includes multiple tokens, such as token1 and token2 corresponding to the black pattern. In this paper, the token can include but is not limited to: words, characters, punctuation marks, special symbols (such as calculation symbols, etc.), etc. In some optional cases, a word can also be referred to as a token, which is not limited in this application.

[0152] Exemplarily, the input data is text or audio information, etc., the processor 140a obtains semantic information of the input data, i.e., a query statement, by performing semantic feature extraction on the input data, and then the processor 140a further performs embedded expression on the query statement to obtain a sequence.

[0153] In other embodiments of the present application, the sequence described above can also be obtained by the computing device 110 processing the input data.

[0154] As the user's demand for the processing capability of the model gradually increases, the number of words in the query statement is increasing, and when a single device is used to deploy the model to process a long sequence, the processing efficiency of the device is low. In the present example, the long sequence refers to a query statement containing a large number of words.

[0155] The inference process of the model will be briefly described below taking the accelerator a 141 as an example. The accelerator a 141 calculates the inference result 1 by calculating the query vector stored locally in the accelerator a 141 and all key-value vectors associated with the query vector in the model inference system 140.

[0156] With respect to possible implementation manners of the query vector (Q), the key vector (K) and the value vector (V), the following will be described taking a word (such as token1) in the query statement as an example: the processor 140a converts the token1 into a vector to obtain the Q, K and V corresponding to the token1. For example, the process of converting the token1 into a vector satisfies the following formulas (1) to (3). Q=XW Q K=XW Q V=XW V

[0157] Wherein, X is an embedding representation corresponding to the token1, i.e., an abstract representation form of mapping the token1 to a low-dimensional space; Q is a query vector, W Q is a weight matrix (weight) corresponding to the query vector, K is a key vector, W K is a weight corresponding to the key vector, V is a value vector, and W V is a weight corresponding to the value vector.

[0158] In this paper, for the sake of simplicity, the key vector and the value vector can be referred to as a key-value vector.

[0159] ​Please continue to refer to FIG. 2, different accelerators are used for different transformer modules in the inference model, and the model parameters of different transformer modules are different, such as the weights stored in different accelerators are different. Since the input data of the model is a query statement, and the model is split into multiple transformer modules, each transformer module includes part of each network layer in the complete transformer. Therefore, the processor 140a inputs the query statement into each transformer module for inference, so as to reduce the amount of calculation to be performed by a single accelerator, improve the calculation efficiency of the accelerators on the input data, and thus improve the inference efficiency of the model.

[0160] For example, the transformer module 1 includes an attention layer 11 and a multi layer perceptron (MLP) 11. The attention layer 11 is part of the complete attention layer in the model, and the attention layer 11 is used to calculate the query vector, the key vector and the value vector corresponding to the input data, and finally output the result of the attention layer 11. The MLP 11 is part of the complete MLP in the model, and the MLP 11 includes two linear layers, which are used to linearly process the result output by the attention layer 11 to obtain the inference result (inference result 1) of the transformer module 1. That is, the transformer module 1 includes part of the weights corresponding to the complete attention layer and part of the weights corresponding to the complete MLP. The contents and inference methods of other transformer modules shown in FIG. 2 can refer to the description of the transformer module 1. For the specific network structure of the transformer module deployed in the accelerator, the following will be exemplarily described in combination with FIG. 2.

[0161] Please refer to FIG. 2, the transformer module includes an encoder and a decoder.

[0162] The encoder is used to process the input data and convert it into an internal representation (e.g., an encoding matrix) that the model can understand. The encoder is stacked by multiple identical layers, and each layer contains two main substructures: a multi-head self-attention mechanism and a feed-forward neural network. As shown in FIG. 2, the encoder includes a multi-head self-attention layer 11, a residual connection and an add and layer normalization (Add&Norm) operation layer (11), a feed-forward layer (11), and an Add&Norm (21). The multi-head self-attention layer 11 is a network processing layer determined based on a multi-head self-attention mechanism, which is used to calculate the dot product between the Q corresponding to the token and the other K and V associated with the Q to obtain the correlation score between each token and other tokens. The feed-forward layer (11) is a network processing layer determined based on a feed-forward neural network, which is used to perform forward feedback on the model parameters. The Add&Norm (11) and the Add&Norm (21) are used to prevent the network layers in the encoder from degenerating and to normalize the activation values of different network layers. For example, the correlation scores determined by the multi-head self-attention layer 11 are weighted and summed to obtain a new representation of each token, such as an encoding matrix.

[0163] The decoder generates an inference result (e.g., inference result 1) based on the encoding matrix generated by the encoder. The encoder includes a mask multi-head self-attention layer 11, an Add&Norm (31), a multi-head self-attention layer 21, an Add&Norm (41), a feed-forward layer (21), an Add&Norm (51), a linear processing layer 11, and a normalization processing layer 11. The functions of the Add&Norm (31) to the Add&Norm (51) can be referred to the description of the aforementioned encoder, which will not be repeated here. The difference between the mask multi-head self-attention layer 11 and the multi-head self-attention layer 21 is that the mask multi-head self-attention layer 11 adopts a masked operation, the KV of the multi-head self-attention layer 21 is calculated using the encoding matrix of the encoder, and the Q is calculated using the output of the previous decoder block. The previous decoder block of the multi-head self-attention layer 21 refers to the decoder corresponding to another token before predicting the token 1. The linear processing layer and the normalization processing layer are used to perform linear processing and normalization operations and output the prediction result corresponding to the token 1, such as the inference result 1.

[0164] It is worth noting that the encoder shown above is a partial network structure of a complete encoder, and the decoder is a partial network structure of a complete decoder. For example, the multi-head self-attention layer 11 is part of the multi-head self-attention layer included in the complete encoder, and the accelerator a 141 stores part of the weight matrix of the complete multi-head self-attention layer, that is, the corresponding part of the weight of the multi-head self-attention layer 11.

[0165] The structure of the transformer model shown in FIG. 2 enables the model to process sequence-to-sequence (seq2seq) tasks, such as machine translation.

[0166] The following shows a possible embodiment of a model inference method that can be applied to the data processing system shown in FIG. 1, for example, to the model inference system 140 in the data processing system. As shown in FIG. 3, which is a flowchart of a model inference method provided by the present application, the model inference method is executed by devices in the model inference system. Among them, the first accelerator can be referred to as accelerator a 141, and the second accelerator can be referred to as accelerator b 142. As shown in FIG. 3, the model inference method can include the following steps S310-S340.

[0167] S310, the model inference system 140 obtains a first inference request.

[0168] In one possible case, the first inference request carries the input data described above, such as a sequence.

[0169] In one possible example, the sequence described above can be a query statement.

[0170] Two possible implementation manners are provided below for the model inference system 140 to obtain the content of the first inference request.

[0171] In one possible implementation manner, the model inference system 140 obtains the first inference request, including: the processor 140a obtains the first inference request.

[0172] In one possible example, the processor 140a obtains the first inference request through the transceiver.

[0173] In one possible case, the processor 140a distributes the received first inference request to the accelerator a 141 and the accelerator b 142.

[0174] In one possible scenario, the processor 140a performs the wire prediction on the first inference request, and determines the required weights to be utilized by the plurality of accelerators in the model inference system when processing the first inference request. The required weights can include a first sub-hot weight in the first hot weight, and a second sub-hot weight in the second hot weight.

[0175] Optionally, the required weights of the plurality of accelerators can further include a first sub-cold weight in the first cold weight, and a second sub-cold weight in the second cold weight. The first cold weight is configured in the accelerator a 141, and the second cold weight is configured in the accelerator b 142.

[0176] Further, the processor 140 sends an identifier of the required weights to each of the plurality of accelerators according to the required weights. The identifier can be a storage address of the weights, or a position of the weights in a weight matrix. The position of the weights in the weight matrix can be directly obtained through the wire prediction.

[0177] For details of this scenario, refer to the description of FIG. 6 below, which will not be repeated here.

[0178] In another possible implementation, the model inference system 140 obtains the first inference request, including: the accelerator a 141 and / or the accelerator b 142 obtains the first inference request.

[0179] In one possible example, the accelerator a 141 is a master accelerator in the model inference system 140, and the master accelerator is responsible for coordinating and integrating the output results of other accelerators (such as the accelerator b 142). Therefore, after obtaining the first inference request, the accelerator a 141 can distribute the first inference request to the accelerator b 142.

[0180] Similarly, in another possible example, the accelerator b 142 is a master accelerator in the model inference system 140, and therefore, after obtaining the first inference request, the accelerator b 142 can distribute the first inference request to the accelerator b 141.

[0181] In yet another possible example, the accelerator a 141 and the accelerator b 142 directly obtain the first inference request.

[0182] For example, the accelerator a 141 and the accelerator b 142 receive the first inference request distributed by the processor 140a.

[0183] S320, the accelerator a 141 processes the first inference request according to the first hot weight, and obtains a first inference result.

[0184] The activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0185] The activation frequency refers to the frequency of use of the weight in the model in the offline prediction or inference process. For example, the processor 140a counts the weight used in processing the input data each time when the model processes different input data, and then takes the ratio of the number of times of each weight to the total number of times of processing the input data by the model as the activation frequency.

[0186] Further, the processor 140a can determine the hot weight and the cold weight (which can be referred to as the cold-hot weight) in the model by using the activation frequency of each weight.

[0187] For details of the processor 140a determining the cold-hot weight in the model by using the activation frequency of each weight, reference can be made to the description of FIG. 5 below, which will not be repeated here.

[0188] In one possible example, the activation frequency of the first hot weight is greater than or equal to the threshold value a, and the activation frequency of the cold weight in the model is less than the threshold value a, so that the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0189] In another possible example, the activation frequency of the first hot weight is greater than the threshold value a, and the activation frequency of the cold weight in the model is less than or equal to the threshold value a, so that the activation frequency of the first hot weight is greater than the activation frequency of the cold weight in the model.

[0190] The threshold value a can be configured by a user.

[0191] In one possible case, the first hot weight is a hot weight in a weight matrix corresponding to one or more network layers in the model.

[0192] For example, the first hot weight is a set of hot weights in a weight matrix corresponding to each network layer in the model.

[0193] In one possible implementation, the accelerator a 141 processes the first inference request according to the first hot weight to obtain the first inference result, including: the accelerator a 141 processes the sequence in the first inference request according to the first hot weight to obtain the first inference result.

[0194] In one possible example, the accelerator a 141 multiplies the first hot weight and the vector corresponding to the sequence to obtain the first inference result. The first inference result is also a vector.

[0195] In a possible implementation, the processor 140a obtains the required utilization of the weights utilized by the plurality of accelerators in the model inference system when processing the first inference request, and the required utilization of the weights includes a first sub-hot weight in the first hot weight. Further, the accelerator a 141 processes the first inference request according to the first hot weight to obtain the first inference result, including: the accelerator a 141 processes the first inference request according to the first sub-hot weight to obtain the first inference result.

[0196] For example, the accelerator a 141 multiplies the first sub-hot weight with the sequence to obtain the first inference result.

[0197] For more possible implementations of S320, refer to the description of FIG. 6 below, which is not repeated here.

[0198] It is worth noting that the first hot weight described above can be stored in a storage space of the data processing system 100, which can be a level 1 cache (L1), a level 2 cache (L2), a level 3 cache (L3), a high bandwidth memory (HBM), etc. of the accelerator a 141, or the storage space is located in a memory connected with the first accelerator.

[0199] S330, the accelerator b 142 processes the first inference request according to the second hot weight to obtain the second inference result.

[0200] In a possible case, the first hot weight and the second hot weight are the same.

[0201] In a possible example, the second hot weight is a hot weight in a weight matrix corresponding to one or more network layers in the model.

[0202] For example, the second hot weight is a set of hot weights in a weight matrix corresponding to each network layer in the model.

[0203] For more details of the second hot weight, refer to the description of the first hot weight above, which is not repeated here.

[0204] In a possible implementation, the accelerator b 142 processes the first inference request according to the second hot weight to obtain the second inference result, including: the accelerator a 141 processes the sequence in the first inference request according to the second hot weight to obtain the second inference result.

[0205] In a possible example, the accelerator b 142 multiplies the second hot weight with a vector corresponding to the sequence to obtain the second inference result. The second inference result is also a vector.

[0206] In a possible implementation, since the processor 140a can obtain the weights required to be utilized when the plurality of accelerators in the model inference system processes the first inference request, and the aforementioned weights required to be utilized include a second sub-hot weight in the second hot weight, further, the accelerator b 142 processes the first inference request according to the second hot weight to obtain the second inference result, including: the accelerator a 141 processes the first inference request according to the second sub-hot weight to obtain the second inference result.

[0207] For example, the accelerator a 141 multiplies the second sub-hot weight with the vector corresponding to the sequence to obtain the second inference result.

[0208] For more possible implementations of S330, refer to the description of FIG. 6 below, which is not described here.

[0209] It is worth noting that the aforementioned second hot weight can be stored in the storage space of the accelerator b 142, which can be L1, L2, L3, HBM, etc. of the first accelerator.

[0210] S340, the accelerator a 141 obtains the output result according to the first inference result and the second inference result.

[0211] In this embodiment, the accelerator a 141 is taken as an example of the main accelerator, and after receiving the second inference result sent by the accelerator b 142, the accelerator a 141 merges the first inference result and the second inference result to obtain the output result.

[0212] In a possible example, if the calculation of the first inference request by the accelerator a 141 using the first hot weight is completely independent of the calculation of the second inference request by the accelerator b 142 using the second hot weight, the accelerator a 141 splices the first inference result and the second inference node.

[0213] For example, the first inference result is a 2x3 matrix a, and the second inference result is a 2x3 matrix b, then the accelerator a 141 splices the matrix a and the matrix b to obtain a 4x3 matrix c.

[0214] In another possible example, if the first network layer corresponding to the first hot weight involves complex calculation, the accelerator a 141 needs to further process when merging the first inference result and the second inference result, such as involving weighting and activation function, etc.

[0215] For example, the first inference result is a 2x3 matrix a, and the second inference result is a 2x3 matrix b. The accelerator a 141 performs splicing on the matrix a and the matrix b after weighting, and still obtains a 4x3 matrix d, and the values in the matrix d are different from the values in the matrix c.

[0216] For example, the first inference result is a 2x3 matrix a, and the second inference result is a 2x3 matrix b. The accelerator a 141 performs splicing on the matrix a and the matrix b to obtain a 4x3 matrix c, and then processes the matrix c by using an activation function to obtain the final output result.

[0217] In a possible case, the accelerator a 141 sends the obtained output result to the terminal device 150.

[0218] In other embodiments of the present application, the main accelerator in the model inference system 140 can be the accelerator b 142, and then the accelerator b 142 obtains the output result according to the first inference result and the second inference result.

[0219] In relation to the content of determining the cold and hot weights in the model by the processor 140a in S320, a possible embodiment is provided as follows.

[0220] The processor 140a obtains the weights required to be used when the model inference system 140 processes a historical inference request, and then obtains the activation frequency of each weight in the model according to the weights required to be used corresponding to each request in a plurality of requests included in the historical inference request.

[0221] In a possible example, as shown in FIG. 4, FIG. 4 is a cold and hot weight determination schematic diagram provided by the present application. The historical inference request includes an inference request 1, an inference request 2, and an inference request 3. The weights required to be used when the model processes the inference request 1 are weight a, weight b, and weight c. The weights required to be used when the model processes the inference request 2 are weight a, weight c, and weight d. The weights required to be used when the model processes the inference request 3 are weight a, weight d, and weight e.

[0222] The processor 140a obtains the activation frequency of each weight according to the weights required to be used corresponding to the inference request. For example, the activation frequency of the weight a is the activation times (3 times) of the weight a divided by the total number of requests (3 times) included in the historical inference request, that is, 100%. Similarly, the activation frequency of the weight b is 1 / 3, that is, 33%. The activation frequency of the weight c is 2 / 3, that is, 67%. The activation frequency of the weight d is 2 / 3, that is, 67%. The activation frequency of the weight e is 1 / 3, that is, 33%.

[0223] Further, the processor 140a determines the weights that are activated in the inference process of each of the historical inference requests as hot weights, such as weight a, weight c, and weight d, and determines the weights that are activated in the inference process of each of the historical inference requests as cold weights, such as weight b and weight e, according to a threshold value a (e.g., 50%) configured by the user.

[0224] In one possible scenario, the processor 140a can further determine the co-occurrence between the weights according to the weights that are required to be utilized by the model when processing the historical inference requests.

[0225] The co-occurrence refers to the frequency of different weights being activated or used together in different inputs of the model. The co-occurrence can be used to understand the relationship between the weights and how they jointly affect the output of the model.

[0226] In one possible example, the processor 140a determines the weights that are activated simultaneously in the inference process of each of the historical inference requests, and then identifies the co-occurrence between the weights through frequent item mining, and stores the combination of the weights with a co-occurrence greater than or equal to a threshold value b (second threshold value) in the same storage unit.

[0227] The combination of the weights can include at least two weights, and all the weights of the model can form multiple combinations of the weights. The storage unit can be a block, cache line, segment, chunk, or the like in the accelerator, or a block, cache line, segment, chunk, or the like in the memory connected to the accelerator.

[0228] In one possible example, the processor 140a can determine the co-occurrence between the weights through frequent item mining by using an association rule algorithm (Apriori), a frequent pattern growth (FP-Growth) algorithm, or the like.

[0229] It is worth noting that the processor 140a determining the hot and cold weights in the model can also be performed by the computing device 110.

[0230] The above describes the processor 140a switching the hot and cold weights according to the weights that are required to be utilized by the model corresponding to the historical inference requests, and the following further provides an embodiment of the model training system 120 training an activation classifier according to the historical inference requests and the weights that are required to be utilized by the model corresponding to the historical inference requests. The activation classifier can be used to process the first inference request to obtain the weights that are required to be utilized by the model when processing the first inference request, thereby realizing the above-described flying line prediction.

[0231] The data processing system 100 trains the activation classifier using the historical inference requests and the corresponding weights of the required utilization, and obtains a trained activation classifier. The activation classifier can establish a mapping relationship between the inference requests and the weights of the required utilization.

[0232] In one possible scenario, the activation classifier can be a logistic regression, a support vector machine (SVM), a decision tree, a random forest, a neural network (such as a multi-layer perceptron (MLP), a convolutional neural network (CNN), etc.), or the like.

[0233] The historical inference requests and the weights of the required utilization when the model processes the historical inference requests can be obtained by the model training system 120 from the database 130. The historical inference requests and the corresponding weights of the required utilization stored in the database 130 can be written by the terminal device 150 or collected by the data collection device 170 and then written.

[0234] As shown in FIG. 5, FIG. 5 is a schematic diagram of training an activation classifier provided by the present application. The model training system 120 trains the activation classifier using the historical inference requests and the corresponding weights of the required utilization. The activation classifier obtained through the foregoing training can actually establish a mapping relationship between the features of the inference requests and the weights of the required utilization.

[0235] The weight of the required utilization includes the position of the weight in the weight matrix.

[0236] For example, the weight a in the complete weight is required when the model processes the historical inference request a, the weight b in the complete weight is required when the model processes the historical inference request b, and the weight c in the complete weight is required when the model processes the historical inference request c.

[0237] For the content of training the activation classifier, conventional technical means in the art can be referred to, and will not be described here.

[0238] It is worth noting that the content of determining the cold and hot weights in the model and training the activation classifier by the processor 140a is an offline preparation process before inference using the model.

[0239] For the content of determining the weights of the required utilization when the plurality of accelerators in the model inference system processes the first inference request by the processor 140a in S310, a possible embodiment is provided as follows.

[0240] The processor 140a inputs the first inference request into the trained activation classifier, outputs the weights to be used in processing the first inference request by the model, and distributes the weights to be used in processing the first inference request to the plurality of accelerators.

[0241] In one possible example, the processor 140a sends the identification of the weights to be used by each accelerator to the corresponding accelerator.

[0242] For example, the processor 140a sends the identification of the weights to be used by the accelerator a 141 to the accelerator a 141. For example, the aforementioned identification includes the storage address of the weight a, the storage address of the weight b, and the first hot weight includes the weight a and the weight d.

[0243] For example, the weight a is located at (1, 2) in the weight matrix corresponding to the first hot weight, and the weight d is located at (2, 3) in the weight matrix corresponding to the first hot weight.

[0244] The processor 140a sends the identification of the weights to be used by the accelerator b 142 to the accelerator b 142. For example, the aforementioned identification includes the storage address of the weight e, the storage address of the weight f, and the second hot weight includes the weight e and the weight f.

[0245] For example, the weight e is located at (1, 2) in the weight matrix corresponding to the first hot weight, and the weight f is located at (2, 3) in the weight matrix corresponding to the first hot weight.

[0246] In the above-mentioned S320, the following provides one possible embodiment. In this embodiment, as shown in FIG. 6, FIG. 6 is a cold and hot weight configuration diagram provided by the present application, the accelerator a 141 has the first hot weight and the first cold weight, and the accelerator b 142 has the second hot weight and the second cold weight.

[0247] As shown in FIG. 6, since the accelerator a 141 and the accelerator b 142 are of the same type, the processor 140a configures the same hot weight (the first hot weight and the second hot weight) into the accelerator a 141 and the accelerator b 142. In order to avoid excessive occupation of the storage resources of each accelerator, the processor 140a divides the determined cold weight into multiple parts and stores them in different accelerators, i.e., one accelerator stores one part of the cold weight. For example, the processor 140a configures the first cold weight for the accelerator a 141 and the second cold weight for the accelerator b 142.

[0248] In one possible case, the accelerator a 141 and the accelerator b 142 are of the same type, which is used to indicate that one or more of the following are the same: the device manufacturer, the product model, the performance, and the like of the accelerator a 141 and the accelerator b 142.

[0249] In one possible example, the processor 140a copies the hot weights into the accelerator a 141 and the accelerator b 142 in the form of broadcasting. The first hot weights can be stored in the HBM of the accelerator a 141, and the second hot weights can be stored in the HBM of the accelerator b 142.

[0250] For example, the high co-occurrence weight combinations in the first hot weights can be stored in the same storage unit in the HBM of the accelerator a 141, such as the same block, the same cache line, the same data segment, or the same chunk, and the like, thereby improving the efficiency of the accelerator a 141 reading the weights in the HBM.

[0251] Similarly, the high co-occurrence weight combinations in the second hot weights can be stored in the same storage unit in the HBM of the accelerator b 142, such as the same block, the same cache line, the same data segment, or the same chunk, and the like, thereby improving the efficiency of the accelerator b 142 reading the weights in the HBM.

[0252] For example, the low co-occurrence weight combinations in the first hot weights can be stored in the same storage unit in the HBM of the accelerator a 141, and the low co-occurrence weight combinations in the second hot weights can be stored in the same storage unit in the HBM of the accelerator b 142.

[0253] Alternatively, the low co-occurrence weight combinations in the first hot weights can be stored in the HBM of the accelerator a 141, and the low co-occurrence weight combinations in the second hot weights can be stored in the HBM of the accelerator b 142.

[0254] For example, the low co-occurrence weight combinations in the first hot weights can be stored in different storage units in the HBM of the accelerator a 141, and the low co-occurrence weight combinations in the second hot weights can be stored in different storage units in the HBM of the accelerator b 142.

[0255] It is worth noting that the high co-occurrence is used to indicate that the co-occurrence is greater than or equal to a threshold b, and the low co-occurrence is used to indicate that the co-occurrence is less than the threshold b.

[0256] It is worth noting that the above is described by taking the accelerator a 141 and the accelerator b 142 as an example, and in other embodiments of the present application, the accelerator a 141 and the accelerator b 142 can also be different, and the first hot weights configured in the accelerator a 141 and the second hot weights configured in the accelerator b 142 are still the same.

[0257] As shown in Figure 7, Figure 7 is a schematic diagram of a hot and cold weight configuration provided in this application. Accelerator a 141 and accelerator b 142 are configured with the same hot weight, and accelerator a 141 and accelerator b 142 are configured with different cold weights.

[0258] For example, processor 140a divides the cold weights in the model into multiple parts using tensor parallelism. Then, processor 140a distributes one set of cold weights to each accelerator. For instance, processor 140a configures a first set of cold weights for accelerator a 141 and a second set of cold weights for accelerator b 142.

[0259] For example, processor 140a vertically or horizontally splits the cold weights of each network layer in the model, and then configures the set of weights obtained after splitting each network layer to the accelerator.

[0260] This application does not limit the number of times the data is split vertically or horizontally. The splitting can be performed according to the number of accelerators so that the number of weighted components obtained after the split matches the number of processors.

[0261] In one possible scenario, the first cold weights include a weight combination 'a' with high co-occurrence, and the second cold weights include a weight combination 'b' with high co-occurrence. The weight combination 'a' with high co-occurrence in the first cold weights is stored in the same storage unit in accelerator a 141, and the weight combination 'b' with high co-occurrence in the second cold weights is stored in the same storage unit in accelerator b 142.

[0262] This application does not limit the number of weight combinations included in the first cold weight and the second cold weight.

[0263] In one possible scenario, the multiple accelerators included in the model inference system 140 are configured with the same thermal weights.

[0264] It is worth noting that, to ensure load balancing across accelerators, the difference in data volume between the first cold weight and the second cold weight is less than or equal to a threshold c, which can be configured by the user. In one possible example, the data volume of the first cold weight and the second cold weight are the same.

[0265] Regarding the details of S320, two possible implementation methods are provided below.

[0266] In one possible implementation, accelerator a 141 obtains all the first hot weights and all the first cold weights from the storage space within accelerator a 141, and then uses all the first hot weights and all the first cold weights to process the first inference request and obtain the first inference result.

[0267] The storage space in the accelerator a 141 can be L1, L2, L3, HBM, etc. in the accelerator a 141.

[0268] For example, the accelerator a 141 multiplies the entire first hot weight and the entire first cold weight with the sequence in the first inference request to obtain the first inference result.

[0269] Similarly, the accelerator b 142 obtains the entire second hot weight and the entire second cold weight from the storage space in the accelerator b 142, and then processes the first inference request by using the entire second hot weight and the entire second cold weight to obtain the second inference result.

[0270] For details of the accelerator b 142 obtaining the second inference result, reference can be made to the description of the accelerator a 141 obtaining the first inference result in the present implementation mode, which will not be repeated here.

[0271] In another possible implementation mode, the accelerator a 141 obtains the first sub-hot weight in the first hot weight and the first sub-cold weight in the first cold weight from the storage space in the accelerator a 141, and then processes the first inference request by using the first sub-hot weight and the first sub-cold weight to obtain the first inference result.

[0272] For example, the accelerator a 141 multiplies the first sub-hot weight and the first sub-cold weight with the sequence to obtain the first inference result.

[0273] It is worth noting that the activation classifier in the processor 140a processes the first inference request to obtain a plurality of weights required to be used by the accelerators to process the first inference request, and the plurality of weights required to be used by the accelerators include the first sub-hot weight and the first sub-cold weight. Then, the processor 140a determines from the correspondence between the weights and the accelerators that the first sub-hot weight and the first sub-cold weight are deployed in the accelerator a 141. Thus, the processor 140a sends the identifier of the first sub-hot weight and the identifier of the first sub-cold weight to the accelerator a 141, and the accelerator a 141 obtains the first sub-hot weight and the first sub-cold weight from the storage space in the accelerator a 141 according to the identifier of the first sub-hot weight and the identifier of the first sub-cold weight.

[0274] Similarly, the accelerator b 142 obtains the second sub-hot weight and the second sub-cold weight from the storage space in the accelerator b 142, and processes the first inference request by using the second sub-hot weight and the second sub-cold weight to obtain the second inference result.

[0275] For details of the accelerator b 142 obtaining the second inference result, reference can be made to the description of the accelerator a 141 obtaining the first inference result in the present implementation mode, which will not be repeated here.

[0276] To achieve load balancing of the accelerator a 141 and the accelerator b 142, the data of the first sub-hot weight and the first sub-cold weight obtained by the accelerator a 141 should be substantially consistent with the data of the second sub-hot weight and the second sub-cold weight obtained by the accelerator b 142, so that the accelerator a 141 and the accelerator b 142 have the same calculation amount and the same load. As shown in FIG. 8, which is a load balancing allocation diagram provided by the present application, the processor 140a will use a load balancing algorithm to determine the weight to be used by each accelerator when processing the first inference request, and then send the identifier of the weight to be used by each accelerator to the corresponding accelerator.

[0277] The processor 140a determines the weight to be used by the model when processing the input data included in the first inference request by activating the classifier, and then inputs the weight to be used into the equalizer. The equalizer determines, by using a load balancing algorithm, that the weight to be used by the accelerator a 141 when processing the first inference request includes the first sub-hot weight and the first sub-cold weight, and the weight to be used by the accelerator b 142 when processing the first inference request includes the second sub-hot weight and the second sub-cold weight.

[0278] The difference between the sum of the data amount of the first sub-hot weight and the data amount of the first sub-cold weight and the sum of the data amount of the second sub-hot weight and the data amount of the second sub-cold weight is less than or equal to a threshold value d (a first threshold value), so as to achieve load balancing of the accelerator a 141 and the accelerator b 142 in the model inference process. The data amount of the weight included in each block in FIG. 8 is consistent.

[0279] For example, the processor 140a determines, by activating the classifier, that the weight to be used by the model when processing the first inference request includes 4 hot weights and 8 cold weights, wherein 4 cold weights are configured in the accelerator a 141 and 4 cold weights are configured in the accelerator b 142.

[0280] Therefore, to ensure load balancing of the accelerator a 141 and the accelerator b 142, the equalizer allocates data with a small difference in calculation amount to the accelerator a 141 and the accelerator b 142 for calculation (for example, the difference between the sum of the data amount of the first sub-hot weight and the data amount of the first sub-cold weight and the sum of the data amount of the second sub-hot weight and the data amount of the second sub-cold weight is less than or equal to the threshold value d). Since the hot weights configured in the accelerator a 141 and the accelerator b 142 are consistent, the processor 140a can instruct the accelerator a 141 to use the above-mentioned 2 hot weights (first sub-hot weight) and 4 cold weights (first sub-cold weight) to calculate the first inference request, and instruct the accelerator b 142 to use the above-mentioned 2 hot weights (second sub-hot weight) and 4 cold weights (second sub-cold weight) to calculate the first inference request.

[0281] For another example, the processor 140a determines that the weights to be utilized by the model in processing the first inference request include 6 hot weights and 6 cold weights, wherein 2 cold weights are configured in the accelerator a 141 and 4 cold weights are configured in the accelerator b 142.

[0282] Therefore, the processor 140a can instruct the accelerator a 141 to perform calculation on the first inference request by utilizing the above-mentioned 4 hot weights (first sub-hot weights) and 2 cold weights (first sub-cold weights), and instruct the accelerator b 142 to perform calculation on the first inference request by utilizing the above-mentioned 2 hot weights (second sub-hot weights) and 4 cold weights (second sub-cold weights).

[0283] One weight in the above-mentioned example can be a weight corresponding to one block in FIG. 8.

[0284] The above-mentioned equalizer can be deployed in the processor 140a.

[0285] In a possible embodiment, on the basis of the model inference method shown in FIG. 3, the present application further provides a cold-hot weight switching method. FIG. 9 is a flowchart of a cold-hot weight switching method provided by the present application. In the present embodiment, the processor 140a maintains a count matrix, which is used to indicate the activation frequency of the weights in the model. As shown in FIG. 9, the cold-hot weight switching method can include the following steps S910 to S940.

[0286] S910, the processor 140a determines a first position of the weights to be utilized by the model inference system 140 in processing a first inference request in the weight matrix;

[0287] In a possible implementation, the processor 140a determines the first position of the weights to be utilized by the model inference system 140 in processing each inference request in the weight matrix.

[0288] The above-mentioned inference request includes the first inference request.

[0289] In a possible case, the first position is the position of the weights to be utilized in the weight matrix, which can be used to determine the second position of the count matrix corresponding to the activation frequency of the weights.

[0290] In a possible example, the processor 140a can process the inference request by activating the classifier to obtain the first position of the weights to be utilized in the weight matrix.

[0291] S920, the processor 140a adds 1 to the value at the second position corresponding to the first position in the count matrix according to the first position.

[0292] In one possible scenario, a tracer in the processor 140a maintains a corresponding count matrix for each weight matrix in the model.

[0293] In one possible example, the count matrix has the same size as the weight matrix, and the first position is the specific position of the required weight in the weight matrix, then the processor 140a adds 1 to the value at the second position in the count matrix corresponding to the first position after determining the first position.

[0294] For example, the count matrix and the weight matrix are both 5x5 matrices, so the positions in the count matrix correspond to the positions in the weight matrix one by one. When the required weight includes weight a and the first position of weight a in the weight matrix is the third row and the fourth column (3, 4), the second position is consistent with the first position, which is also (3, 4), so the processor 140a adds 1 to the value at (3, 4) in the count matrix.

[0295] In another possible example, the count matrix has the same number of columns as the weight matrix, the first position is the column where the required weight is located in the weight matrix, and the count matrix has only one row, then the processor 140a adds 1 to the value at the second position in the count matrix corresponding to the first position after determining the first position.

[0296] For example, the weight matrix is a 5x5 matrix, and the count matrix is a 1x5 matrix, so a column in the weight matrix corresponds to a position in the weight matrix. When the required weight includes weight a and the first position of weight a in the weight matrix is (3, 4), the second position is consistent with the column number of the first position, so the second position is (1, 4), and then the processor 140a adds 1 to the value at (1, 4) in the count matrix.

[0297] In yet another possible example, the count matrix has the same number of columns as the number of rows of the weight matrix, the first position is the row where the required weight is located in the weight matrix, and the count matrix has only one row, then the processor 140a adds 1 to the value at the second position in the count matrix corresponding to the first position after determining the first position.

[0298] For example, the weight matrix is a 5x5 matrix, and the count matrix is a 1x5 matrix, so a row in the weight matrix corresponds to a position in the weight matrix. When the required weight includes weight a and the first position of weight a in the weight matrix is (3, 4), the number of rows of the first position is consistent with the number of columns of the second position, so the second position is (1, 3), and then the processor 140a adds 1 to the value at (1, 3) in the count matrix.

[0299] S930, if the value at the third position in the count matrix is greater than or equal to the threshold value e (the third threshold value), the processor 140a sends a first weight switching request to the accelerator a 141.

[0300] The first weight switching request is used to instruct switching of the first cold weight configured in the first accelerator.

[0301] The weight corresponding to the third position is configured in the accelerator a 141, so the tracker in the processor 140a determines that the value at the fourth position in the count matrix is greater than or equal to the threshold value e, and sends the first weight switching request to the accelerator a 141.

[0302] In a possible example, if the count matrix is consistent with the size of the weight matrix, the weight corresponding to the third position is a single weight. For example, if the third position is (3, 4), the weight corresponding to the third position is the weight at the position (3, 4) in the weight matrix.

[0303] In another possible example, if the count matrix is consistent with the number of columns of the weight matrix, the weight corresponding to the third position is a column weight. For example, if the third position is (1, 2), the weight corresponding to the third position is the weight corresponding to the second column in the weight matrix.

[0304] In yet another possible example, if the number of columns of the count matrix is consistent with the number of rows of the weight matrix, the weight corresponding to the third position is a row weight. For example, if the third position is (1, 2), the weight corresponding to the third position is the weight corresponding to the second row in the weight matrix.

[0305] In a possible implementation, if the data at the third position in the count matrix is greater than or equal to the threshold value e (such as 50), the processor 140a sends a first weight switching request to the accelerator a 141, which is used to instruct switching of the third sub-cold weight (including the single weight, the column weight, and the row weight).

[0306] The data at the third position being greater than or equal to the threshold value e indicates that the activation frequency of the third sub-cold weight corresponding to the third position is greater than the activation frequency of the weights other than the third sub-cold weight in the cold weight.

[0307] It is worth noting that the above only takes the weight corresponding to the third position as an example for description, and in other embodiments of the present application, the weight corresponding to the third position can also be configured in other accelerators in the model inference system 140.

[0308] S940, the accelerator a 141 switches the third sub-cold weight in the first cold weight to a hot weight according to the first weight switching request.

[0309] The activation frequency of the third sub cold weight is greater than the activation frequency of other weights in the cold weight except the third sub cold weight.

[0310] In a possible case, the distance between the storage area of the first hot weight and the acceleration chip in the first accelerator is less than or equal to the distance between the storage area of the first cold weight and the acceleration chip, and therefore, the accelerator a 141 migrates the third sub cold weight from the storage area of the first cold weight to the storage area of the first hot weight according to the first weight switching request, so as to realize the third sub cold weight as a hot weight.

[0311] In a possible example, the storage area of the first hot weight and the storage area of the first cold weight are both located in the HBM in the accelerator a 141, and the accelerator a 141 can copy the third sub cold weight to the buffer of the HBM, and then copy the third sub cold weight in the buffer to the storage area of the first hot weight, so as to realize the migration of the third sub cold weight from the storage area of the first cold weight to the storage area of the first hot weight, thereby realizing the online migration of the cold and hot weights.

[0312] In a possible case, after the accelerator a 141 switches the third sub cold weight in the first cold weight to a hot weight, the accelerator a 141 sends a response of switching completion to the processor 140a, and the processor 140a restores the count state of the count matrix maintained by the tracker according to the response, that is, clears the count matrix.

[0313] It is worth noting that since the plurality of accelerators included in the model inference system 140 are all configured with the same hot weight, when the accelerator a 141 performs cold weight switching, other accelerators in the model inference system 140 also need to perform hot weight switching. As shown in FIG. 9, since the third sub cold weight is stored in the storage space in the accelerator a 141, the third sub cold weight can be transmitted between the accelerators through the high-speed interconnection bandwidth, so as to realize that other accelerators take the third sub cold weight as a hot weight.

[0314] Taking the accelerator b 142 in other accelerators and the third sub cold weight stored in the HBM in the accelerator a 141 as an example, the accelerator a 141 copies the third sub cold weight to the buffer of the HBM, synchronizes the third sub cold weight in the global range (all accelerators) through the high-speed interconnection bandwidth between the accelerators, and then the accelerator b 142 switches the third sub cold weight to a hot weight.

[0315] For example, the accelerator a 141 performs ALL2ALL communication through the high-speed interconnection bandwidth, synchronizes the data (the third sub cold weight) in the global range, and the aforementioned data exchange is completed in the buffer, so as to realize the online cold and hot weight switching.

[0316] The accelerator a 141 synchronizes the third sub-cold weight stored in the buffer in the accelerator a 141 to the buffer in the accelerator b 142 through the high-speed interconnection bandwidth between the accelerator a 141 and the accelerator b 142, and then the accelerator b 142 can store the third sub-cold weight in the buffer to the storage area corresponding to the first hot weight, so as to realize that the accelerator b 142 takes the third sub-cold weight as the hot weight.

[0317] For example, the high-speed interconnection bandwidth between the above accelerators can be RDMA (remote direct memory access) and the like. The RDMA can include IB (infiniband), RoCE (RDMA over converged ethernet), iWARP (internet wide area RDMA protocol) and the like. The IB is a high-performance interconnection technology that provides native RDMA support. The RoCE is a technology that implements RDMA over Ethernet. It uses standard Ethernet as the underlying transport medium and implements RDMA functionality by using RoCE adapters and appropriate protocol stacks. The iWARP is an RDMA implementation based on the TCP / IP protocol stack. It uses a common Ethernet adapter and a standard network switch, and provides high-performance remote memory access and data transmission by implementing RDMA functionality in the TCP / IP protocol stack.

[0318] It is worth noting that the above content should not be understood as a limitation of the present application. In another embodiment of the present application, if the value at the fourth position in the count matrix is less than the threshold value f (the fourth threshold value), the processor 140a sends a second weight switching request to the accelerator a 141. The accelerator a 141 switches the third sub-hot weight in the first hot weight to a cold weight according to the second weight switching request. The second switching request is used to indicate switching of the first hot weight configured in the accelerator a 141, and the activation frequency of the third sub-hot weight is less than the activation frequency of the other weights in the first hot weight except the third sub-hot weight.

[0319] For the content of the count matrix, reference can be made to the description in the above embodiment of the cold-hot weight switching, which will not be repeated here.

[0320] The weight corresponding to the fourth position is the third sub-hot weight (a single weight, a column of weights, a row of weights), and the value at the fourth position is less than the threshold value f, which means that the activation frequency of the third sub-hot weight corresponding to the fourth position is less than the activation frequency of the other weights in the first hot weight except the third sub-hot weight.

[0321] In a possible case, the storage area of the first hot weight is closer to the acceleration chip in the first accelerator than the storage area of the first cold weight, and therefore, the accelerator a 141 migrates the third sub-hot weight from the storage area of the first hot weight to the storage area of the first cold weight according to the second weight switching request, so as to switch the third sub-hot weight to a cold weight.

[0322] In a possible example, the storage area of the first hot weight and the storage area of the first cold weight are both located in the HBM in the accelerator a 141, and the accelerator a 141 can copy the third sub-hot weight to a buffer of the HBM, and then copy the third sub-hot weight in the buffer to the storage area of the first cold weight, so as to migrate the third sub-hot weight from the storage area of the first hot weight to the storage area of the first cold weight, thereby realizing the online migration of the cold and hot weights.

[0323] It is worth noting that, since the plurality of accelerators included in the model inference system 140 are all configured with the same hot weight, when the accelerator a 141 performs the cold weight switching, the other accelerators in the model inference system 140 also need to perform the cold and hot weight switching. For the content of the cold and hot weight switching of the other accelerators, reference can be made to the description in the above embodiments of the cold and hot weight switching, which will not be repeated here.

[0324] In another embodiment of the present application, if the value at the third position in the counting matrix is greater than the threshold value e and the value at the fourth position in the counting matrix is less than the threshold value f, a third weight switching request is sent to the accelerator a 141. The accelerator a 141 switches the third sub-hot weight in the first hot weight to a cold weight and switches the third sub-cold weight in the first cold weight configured in the first accelerator to a hot weight according to the third weight switching request. The third weight switching request is used to indicate the switching of the weights configured in the first accelerator. The activation frequency of the third sub-hot weight is less than the activation frequency of the other weights in the first hot weight except the third sub-hot weight, and the activation frequency of the third sub-cold weight is greater than the activation frequency of the other weights in the cold weight except the third sub-cold weight.

[0325] For the details of the present embodiment, reference can be made to the description of the above two possible embodiments of the cold and hot weight switching, which will not be repeated here.

[0326] In order to complete and clearly illustrate the overall flow of the model inference method, a possible embodiment is provided as follows. As shown in FIG. 10, FIG. 10 is a flowchart of a model inference method provided by the present application. The method shown in FIG. 10 includes the following steps S1010-S1050.

[0327] S1010, the model training system 120 trains the activated classifier. S1020, the model training system 120 trains the activated classifier. S1030, the model training system 120 trains the activated classifier. S1040, the model training system 120 trains the activated classifier. S1050, the model training system 120 trains the activated classifier.

[0328] For details of S1010, refer to the content shown in FIG. 5 described above, which will not be repeated here.

[0329] S1020, the model inference system 140 determines the cold-hot weight in the model, and deploys the same hot weight and different cold weights for each accelerator.

[0330] For details of S1020, refer to the content shown in FIG. 4, FIG. 6, and FIG. 7 described above, which will not be repeated here.

[0331] S1030, the model inference system 140 determines the required weight for the plurality of accelerators to process the first inference request by using the activation classifier, and the plurality of accelerators processes the first inference request according to the required weight to obtain an output result.

[0332] For details of S1030, refer to the content shown in FIG. 3 and FIG. 8 described above, which will not be repeated here.

[0333] S1040, the model inference system 140 counts the activation frequency / activation times of the weight during processing the first inference request.

[0334] S1050, the model inference system 140 switches the cold-hot weight in the model according to the activation frequency / activation times of the weight.

[0335] For details of S1040 and S1050, refer to the content shown in FIG. 9 described above, which will not be repeated here.

[0336] The above content shows a homogeneous model inference system, and in a possible embodiment, the above model inference system can also be heterogeneous. For example, the plurality of accelerators included in the model inference system 140 can further include a third accelerator, which is different from the types of the first accelerator and the second accelerator.

[0337] In a possible case, the third accelerator is configured with a third hot weight of the model, which is different from the first hot weight.

[0338] In this case, the hot weight in the model is divided into multiple parts, such as two parts, one of which is the first hot weight or the second hot weight, and the other of which is the third hot weight.

[0339] For example, the hot weight corresponding to each network layer in the model is divided into two parts, the first hot weight includes one part of the hot weight corresponding to each network layer in the model, and the third hot weight includes the other part of the hot weight corresponding to each network layer in the model.

[0340] In a possible implementation, the model inference method further includes: the third accelerator processes the first inference request according to the third hot weight to obtain a third inference result. The first accelerator obtains an output result according to the first inference result and the second inference result, including: the first accelerator obtains the output result according to the first inference result, the second inference result, and the third inference result.

[0341] For the content that the first accelerator obtains the output result according to the first inference result, the second inference result, and the third inference result, refer to the description of the first accelerator obtaining the output result according to the first inference result and the second inference result, which is not repeated here.

[0342] In another possible case, the third accelerator is configured with a third hot weight of the model, and the third hot weight is the same as the first hot weight.

[0343] In this case, there is only one copy of the hot weight in the model, and each accelerator stores the same copy of the hot weight.

[0344] For the content of this case, refer to the content of the above case, which is not repeated here.

[0345] The above shows an embodiment in which the first hot weight and the second hot weight are the same. In a possible embodiment, the first hot weight and the second hot weight can be different.

[0346] That is, the hot weight in the model is divided into multiple copies, such as two copies, one of which is the first hot weight and the other of which is the second hot weight.

[0347] For example, the hot weight corresponding to each network layer in the model is divided into two copies, the first hot weight includes one copy of the hot weight corresponding to each network layer in the model, and the second hot weight includes another copy of the hot weight corresponding to each network layer in the model.

[0348] For the content of this embodiment, refer to the description of the embodiment in which the first hot weight and the second hot weight are the same, which is not repeated here.

[0349] It can be understood that, in order to implement the functions in the above embodiments, the model inference system includes hardware structures and / or software modules corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application scenario and design constraints of the technical solution.

[0350] The chip provided in the embodiments of the present application also includes a memory for storing program instructions and / or data.

[0351] In a possible design, the chip further includes a memory for storing program instructions and / or data.

[0352] The chip provided in the embodiments of the present application can implement the functions of the accelerator a 141 or the accelerator b 142, and details are not described herein again.

[0353] The server provided in the embodiments of the present application includes a processor, a plurality of accelerators and a memory. The plurality of accelerators include a first accelerator and a second accelerator. The processor is configured to execute instructions stored in the memory to obtain a first inference request. The first accelerator is configured to implement the function of the accelerator a 141 in the method embodiments, and the second accelerator is configured to implement the function of the accelerator b 142 in the method embodiments.

[0354] That is, the server provided in the embodiments of the present application is a supernode. The supernode refers to a plurality of GPUs interconnected through a bus, which appears as a larger server.

[0355] The chip provided in the embodiments of the present application can implement the functions of the accelerator a 141 or the accelerator b 142, and details are not described herein again.

[0356] The accelerator provided in the embodiments of the present application is described below with reference to the accompanying drawings. FIG. 11 is a structural schematic diagram of an accelerator provided in the present application. The accelerator 1100 can be used to implement the function of the accelerator in the method embodiments, and thus can also implement the beneficial effects of the method embodiments. In this embodiment, the accelerator 1100 can be the accelerator a 141 as shown in FIG. 1, that is, the first accelerator.

[0357] As shown in FIG. 11, the accelerator 1100 can include an acceleration chip 1120. Optionally, the accelerator 1100 can also include a memory 1130 and / or a transceiver 1110. Among others, the acceleration chip 1120 is coupled with the memory 1130 and the transceiver 1110, such as can be connected through a communication bus. The communication bus can include, but is not limited to, a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0358] The various components of the accelerator 1100 will be described in detail below in conjunction with FIG. 11.

[0359] Among others, the acceleration chip 1120 is a computing unit of the accelerator 1100, which can be a control circuit or a collective term of a plurality of processing elements. For example, the acceleration chip 1120 is one or more GPUs, TPUs, NPUs, or ASICs, or one or more integrated circuits configured to implement embodiments of the present application, such as one or more DSPs, or one or more FPGAs.

[0360] Optionally, the acceleration chip 1120 can perform various functions of the accelerator 1100 by running or executing software programs stored in the memory 1130 and invoking data stored in the memory 1130. In a specific implementation, as an embodiment, the acceleration chip 1120 can include one or more GPUs.

[0361] Optionally, the accelerator 1100 can also include a plurality of acceleration chips. Each of these acceleration chips can be a single-GPU or a multi-GPU. Here, the acceleration chip can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0362] The memory 1130 is configured to store a software program for performing model inference in the scheme of the present application, and the execution is controlled by the acceleration chip 1120. The specific implementation can refer to the model inference in the method embodiments described above, and details are not described herein. For example, the memory 1130 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1130 can be integrated with the acceleration chip 1120, or can exist independently and be coupled to the acceleration chip 1120 through an interface circuit (not shown in FIG. 11) of the accelerator 1100, and the embodiments of the present application are not limited in this regard.

[0363] The transceiver 1110 is configured to communicate with other devices. For example, the accelerator 1100 is a client or an application server, and the transceiver 1110 can be configured to communicate with the acceleration chip or another processing chip. For another example, the accelerator 1100 is a multi-core chip, and the transceiver 1110 can be configured to communicate with another multi-core chip.

[0364] Optionally, the transceiver 1110 can include a receiver and a transmitter (not shown separately in FIG. 11). The receiver is configured to implement the receiving function, and the transmitter is configured to implement the transmitting function. Optionally, the transceiver 1110 can be integrated with the acceleration chip 1120, or can exist independently and be coupled to the acceleration chip 1120 through an interface circuit (not shown in FIG. 11) of the accelerator 1100, and the embodiments of the present application are not limited in this regard.

[0365] In this embodiment, the transceiver 1110 is configured to obtain the first inference request. The acceleration chip 1120 processes the first inference request according to the first thermal weight, to obtain a first inference result, and obtains an output result according to the first inference result and a second inference result. The accelerator 1100 processes the first inference request in cooperation with a second accelerator, which processes the first inference request according to a second thermal weight, to obtain the second inference result. Therefore, the transceiver 1110 is further configured to receive the second inference result sent by the second accelerator. The accelerator 1100 can be configured to implement the functions of the accelerator a 141 and the corresponding advantages in the foregoing embodiments, and the second accelerator can be configured to implement the functions of the accelerator b 142 and the corresponding advantages in the foregoing embodiments, which will not be described herein.

[0366] In a possible case, for the structure of the second accelerator, reference can be made to the description of the accelerator 1100, which will not be described herein.

[0367] The method steps in this embodiment can be implemented by means of hardware, or by means of software instructions executed by a processor. The software instructions can be composed of corresponding software modules, which can be stored in a RAM, a flash memory, a ROM, a PROM, an EPROM, an EEPROM, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0368] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable apparatus. The computer programs or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer programs or instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired or wireless means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc (digital video disc, DVD); and a semiconductor medium, such as a solid state drive (solid state drive, SSD).

[0369] Notably, if the computer program product can be provided to users for use through a cloud service subscription mode, users can choose different subscription levels according to their needs; for example, the computer program product can also provide enterprise-level customization services with professional domain customization, interface personalization and expansion functions according to the needs of users or enterprises. In addition, the computer program product provided by the present application which can implement the above model training method can also be made into value-added services for users, which is not limited by the present application. Alternatively, the computer program product can also be embedded in the eDataMateTM or other large language model (large model) tool chain system, such as operator acceleration library, etc.

[0370] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A model inference method, comprising: The method is applied to a model inference system, the model inference system comprising a plurality of accelerators, the plurality of accelerators comprising a first accelerator and a second accelerator, the first accelerator being configured with a first hot weight of a model, the second accelerator being configured with a second hot weight of the model; the method comprising: The model inference system obtains a first inference request; The first accelerator processes the first inference request according to the first hot weight to obtain a first inference result; the activation frequency of the first hot weight is greater than the activation frequency of a cold weight in the model; The second accelerator processes the first inference request according to the second hot weight to obtain a second inference result; the second hot weight is the same as the first hot weight; The first accelerator obtains an output result according to the first inference result and the second inference result.

2. The method of claim 1, wherein, The first accelerator and the second accelerator are of the same type.

3. The method according to claim 1 or 2, characterized in that, The plurality of accelerators further comprises a third accelerator, the third accelerator being different from the first accelerator and the second accelerator in type; the third accelerator is configured with a third hot weight of the model, the third hot weight being different from the first hot weight; The method further comprises: The third accelerator processes the first inference request according to the third hot weight to obtain a third inference result; The first accelerator obtains an output result according to the first inference result and the second inference result, comprising: The first accelerator obtains the output result according to the first inference result, the second inference result and the third inference result.

4. The method according to any one of claims 1 to 3, characterized in that, The first accelerator is further configured with a first cold weight in the cold weight, and the second accelerator is further configured with a second cold weight in the cold weight; The first accelerator processes the first inference request according to the first hot weight to obtain a first inference result, comprising: The first accelerator processes the first inference request according to the first hot weight and the first cold weight to obtain the first inference result; The second accelerator processes the first inference request according to the second hot weight to obtain a second inference result, comprising: The second accelerator processes the first inference request according to the second hot weight and the second cold weight to obtain the second inference result.

5. The method of claim 4, wherein, The first accelerator processes the first inference request according to the first hot weight and the first cold weight to obtain the first inference result, comprising: The first accelerator processes the first inference request according to a first sub-hot weight in the first hot weight and a first sub-cold weight in the first cold weight to obtain the first inference result; The second accelerator processes the first inference request according to the second hot weight and the second cold weight to obtain the second inference result, comprising: The second accelerator processes the first inference request according to a second sub-hot weight in the second hot weight and a second sub-cold weight in the second cold weight to obtain the second inference result.

6. The method of claim 5, wherein, A difference between a sum of a data amount of the first sub-hot weight and a data amount of the first sub-cold weight and a sum of a data amount of the second sub-hot weight and a data amount of the second sub-cold weight is less than or equal to a first threshold value.

7. The method according to any one of claims 4 to 6, characterized in that, The first cold weight includes a weight combination with a co-occurrence degree greater than or equal to a second threshold value, and the weight combination is stored in a same storage unit in the first accelerator.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The first accelerator obtains a first weight switching request; the first weight switching request is used to indicate switching of the first cold weight configured in the first accelerator; The first accelerator switches a third sub-cold weight in the first cold weight to a hot weight according to the first weight switching request; an activation frequency of the third sub-cold weight is greater than an activation frequency of other weights in the cold weight except the third sub-cold weight.

9. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The first accelerator obtains a second weight switching request; the second weight switching request is used to indicate switching of the first hot weight configured in the first accelerator; The first accelerator switches a third sub-hot weight in the first hot weight to a cold weight according to the second weight switching request; an activation frequency of the third sub-hot weight is less than an activation frequency of other weights in the first hot weight except the third sub-hot weight.

10. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The first accelerator obtains a third weight switching request; the third weight switching request is used to indicate switching of the weight configured in the first accelerator; The first accelerator switches a third sub-hot weight in the first hot weight to a cold weight and switches a third sub-cold weight in the first cold weight configured in the first accelerator to a hot weight according to the third weight switching request; the activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weight except the third sub-hot weight, and the activation frequency of the third sub-cold weight is greater than the activation frequency of other weights in the cold weight except the third sub-cold weight.

11. A model inference system comprising: The model inference system includes a plurality of accelerators, the plurality of accelerators including a first accelerator and a second accelerator, the first accelerator being configured with a first hot weight of a model, and the second accelerator being configured with a second hot weight of the model; The model inference system is configured to obtain a first inference request; The first accelerator is configured to process the first inference request according to the first hot weight to obtain a first inference result; The activation frequency of the first hot weight is greater than an activation frequency of a cold weight in the model; The second accelerator is configured to process the first inference request according to the second hot weight to obtain a second inference result; the second hot weight is the same as the first hot weight; The first accelerator is configured to obtain an output result according to the first inference result and the second inference result.

12. The system of claim 11, wherein, The first accelerator and the second accelerator are of the same type.

13. The system of claim 11 or 12, wherein, The plurality of accelerators further comprises a third accelerator, which is different from the types of the first and second accelerators; the third accelerator is configured with a third hot weight of the model, which is different from the first hot weight; The third accelerator is configured to process the first inference request according to the third hot weight to obtain a third inference result; The first accelerator is specifically configured to obtain the output result according to the first inference result, the second inference result and the third inference result.

14. The system of any one of claims 11 to 13, wherein, The first accelerator is further configured with a first cold weight of the cold weights, and the second accelerator is further configured with a second cold weight of the cold weights; The first accelerator is specifically configured to process the first inference request according to the first hot weight and the first cold weight to obtain the first inference result; The second accelerator is specifically configured to process the first inference request according to the second hot weight and the second cold weight to obtain the second inference result.

15. The system of claim 14, wherein, The first accelerator is specifically further configured to process the first inference request according to a first sub-hot weight of the first hot weight and a first sub-cold weight of the first cold weight to obtain the first inference result; The second accelerator is specifically further configured to process the first inference request according to a second sub-hot weight of the second hot weight and a second sub-cold weight of the second cold weight to obtain the second inference result.

16. The system of claim 15, wherein, The difference between the sum of the data amount of the first sub-hot weight and the data amount of the first sub-cold weight and the sum of the data amount of the second sub-hot weight and the data amount of the second sub-cold weight is less than or equal to a first threshold value.

17. The system of any one of claims 14 to 16, wherein, The first cold weight comprises a weight combination with a co-occurrence degree greater than or equal to a second threshold value, and the weight combination is stored in the same storage unit in the first accelerator.

18. The system of any one of claims 11 to 17, wherein, The first accelerator is further configured to obtain a first weight switching request; the first weight switching request is used to indicate switching of the first cold weight configured in the first accelerator; The first accelerator is further configured to switch a third sub-cold weight in the first cold weight to a hot weight according to the first weight switching request; the activation frequency of the third sub-cold weight is greater than the activation frequency of other weights in the cold weight except the third sub-cold weight.

19. The system of any one of claims 11 to 17, wherein, The first accelerator is further configured to obtain a second weight switching request; the second weight switching request is used to indicate switching of the first hot weight configured in the first accelerator; The first accelerator is further configured to switch a third sub-hot weight in the first hot weight to a cold weight according to the second weight switching request; the activation frequency of the third sub-hot weight is less than the activation frequency of other weights in the first hot weight except the third sub-hot weight.

20. The system of any one of claims 11 to 17, wherein, The first accelerator is further configured to obtain a third weight switching request; the third weight switching request is used to indicate switching of the weight configured in the first accelerator; The first accelerator is further configured to switch a third sub-hot weight in the first hot weight to a cold weight and switch a third sub-cold weight in a first cold weight configured in the first accelerator to a hot weight according to the third weight switching request; an activation frequency of the third sub-hot weight is less than that of other weights in the first hot weight except the third sub-hot weight, and an activation frequency of the third sub-cold weight is greater than that of other weights in the cold weight except the third sub-cold weight.

21. A chip, characterized by The chip comprises a control circuit and an interface circuit; The interface circuit is configured to obtain a first inference request, and the control circuit is configured to perform processing of the first inference request by the first accelerator or the second accelerator in the method according to any one of claims 1 to 10.

22. A server, characterized by The system comprises a processor, a plurality of accelerators and a memory; The processor is configured to execute instructions stored in the memory to obtain a first inference request; A first accelerator in the plurality of accelerators is configured with a first hot weight of a model, and the first accelerator processes the first inference request according to the first hot weight to obtain a first inference result; an activation frequency of the first hot weight is greater than that of a cold weight in the model; A second accelerator in the plurality of accelerators is configured with a second hot weight of the model, and the second accelerator processes the first inference request according to the second hot weight to obtain a second inference result; the second hot weight is the same as the first hot weight; The first accelerator obtains an output result according to the first inference result and the second inference result.

23. A cluster of computing devices, characterized in that, The system comprises at least one computing device, each computing device comprising an accelerator and a memory; The accelerator of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform processing of the first inference request by the first accelerator or the second accelerator in the method according to any one of claims 1 to 10.

Images