Expert access prediction method and system suitable for hybrid expert architecture large language model

Abstract

The application provides a method and system for predicting expert access of a mixed expert architecture large language model, which comprehensively considers model structure and hardware feature information, constructs a neural network-based expert access predictor for each layer, and trains the predictor using historical data to predict the experts that need to be activated in the model inference process in advance. In this process, the predictor performs prediction in parallel on the CPU, and the prefetcher loads the expert parameters from the host memory to the GPU memory in advance according to the prediction result. By adjusting the prediction distance, the prediction accuracy and the prefetching advance amount are balanced, and the prefetching amount can be dynamically adjusted according to the prediction accuracy. At the same time, parallel loading is adopted to maximize the use of system resources and reduce the impact on GPU inference performance. Finally, mechanisms such as high-priority re-prefetching are used to handle prediction errors to ensure normal model inference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model prediction, in particular to an expert access prediction method and system suitable for a large language model with a mixture-of-experts architecture. BACKGROUND

[0002] With the wide application of large language models (LLMs) in natural language processing, content generation, and decision support, there is an increasing demand for running LLMs on consumer-grade platforms to achieve better privacy protection and response speed. However, due to the large memory requirements of LLMs, which often exceed the capacity of consumer-grade GPUs, this memory limitation severely affects the performance and application of LLMs on personal computers.

[0003] Mixture-of-Experts (MoE) provides an opportunity to address GPU memory limitations by dividing the model into multiple experts and only activating a subset of them during inference. This allows most expert parameters to be offloaded to the host memory and only load the necessary experts into the GPU memory when needed. Although this significantly reduces the GPU memory requirement, expert offloading also causes a significant performance degradation due to the limited PCIe bandwidth between the host memory and the GPU.

[0004] To address this issue, existing systems propose a solution to cache frequently accessed expert parameters in the GPU memory to minimize the offloading overhead. However, the cache handles cache misses in a passive manner. When the inference process encounters a missing expert in the GPU memory, the computation is blocked until the expert is fetched from the host memory. This leaves a high latency overhead on the critical path of inference, causing a significant impact on performance and utilization.

[0005] Another optimization approach is to predict which experts will be used next and prefetch them in advance. This method can theoretically significantly improve efficiency, as it can overlap the loading of expert parameters with the computation process, thereby hiding the latency of data transfer. However, due to the difficulty in grasping the accuracy and advance of prediction, this method often performs poorly in practice. Low prediction accuracy can lead to a large number of invalid prefetches, not only wasting bandwidth resources but also potentially interfering with the loading of truly needed experts; and too small prediction advance can result in insufficient time for prefetching to be completed before the expert is accessed. It is worth noting that these problems manifest differently under different model architectures and application scenarios.

[0006] Therefore, it is desirable to have a solution that can accurately predict the expert to be accessed and reasonably arrange the prefetch timing, while adapting to different model structures and hardware platforms. Specifically, an expert access prediction mechanism for a hybrid expert architecture is needed, which can effectively combine model structure and runtime features to improve prediction accuracy, and dynamically adjust the prefetch strategy according to hardware characteristics to maximize the prefetch effect and improve model inference speed.

[0007] Patent application document CN118761472A discloses a hybrid expert model inference acceleration method, device, equipment, medium and program, wherein the method comprises: loading target correlation parameters of a hybrid expert model; wherein the target correlation parameters comprise non-expert network parameters and expert network benchmark vectors of the hybrid expert model; predicting a target expert network of a to-be-run network layer of the hybrid expert model according to a current layer output feature vector of a current running network layer of the hybrid expert model and the expert network benchmark vector of the to-be-run network layer; and preloading the target expert network of the to-be-run network layer to a device storage for running the hybrid expert model to perform a model inference process. However, this patent cannot completely solve the existing technical problems and cannot meet the needs of the present application. SUMMARY

[0008] In view of the defects in the prior art, the purpose of the present application is to provide an expert access prediction method and system suitable for a hybrid expert architecture large language model.

[0009] The expert access prediction method suitable for a hybrid expert architecture large language model provided by the present application comprises:

[0010] Step 1: Obtain the structure information of the hybrid expert architecture large language model and the hardware feature information of the computing platform, including the number of model layers, the number of experts per layer, the number of experts to be activated for each word per layer, the number of parameters of each expert, the PCIe link bandwidth and the available GPU memory size;

[0011] Step 2: Construct a neural network-based expert access predictor P for each layer of the large language model i The predictor is responsible for predicting the experts to be activated according to the input before the kth layer, wherein P i is the predictor of the ith layer, which consists of a double-layer perceptron, the input h i-k is the input of the i-kth layer of the large language model, and the output e i is the probability of each expert of the ith layer being activated, and k is a prediction distance specified by the user in advance;

[0012] Step 3: Collect historical data in the inference process of the large language model as training data, and for each inference process of a word, record the input of each layer gating function and the actually selected expert;

[0013] Step 4: Offline training of the expert access predictor for each layer using the collected training data, and persisting to disk;

[0014] Step 5: Starting the inference service, loading the non-expert parameters into GPU memory, loading the expert parameters into CPU memory, and loading the corresponding predictor for each layer of the large language model into CPU memory;

[0015] Step 6: During online inference, input the gate function of each layer in the GPU inference process to the CPU memory, and the predictor corresponding to the collected input calculates e i = P i (h i-k ), output the probability of each expert being used after k layers, as the prediction result;

[0016] Step 7: According to the prediction result, select the same number of experts as the number of model active experts for each layer and put them into the prefetch queue, and hand them over to the prefetcher to perform prefetching. The prefetcher loads the expert parameters from the host memory to the GPU memory according to the order of entry into the queue;

[0017] Step 8: When the inference reaches the predicted layer, the experts that have not been prefetched due to prediction errors are put back into the prefetch queue and handed over to the prefetcher to perform prefetching;

[0018] Step 9: After the inference of a word is completed, the inference result is returned to the user, and the process returns to Step 6 to start the inference of the next word.

[0019] Preferably, in Step 4, the expert access predictor for each layer is offline trained, specifically: for each input in the training data, compare the output of the predictor with the actual output, calculate the loss function, and use backpropagation with gradient descent method to update the predictor parameters. Repeat the process of traversing the training data until the predictor converges.

[0020] Preferably, in Step 6, the collection process and the prediction process are performed on the CPU in parallel with the GPU inference process.

[0021] Preferably, in Step 7, the expert parameter loading process is performed in parallel with the GPU inference process, and the number of prefetched experts is dynamically adjusted according to the accuracy of the predictor.

[0022] Preferably, in Step 8, the re-prefetching caused by prediction errors enters the prefetch queue with high priority, and the unfinished error prefetching tasks in the queue are cleared.

[0023] The application provides a specialist access prediction system suitable for a mixed specialist architecture large language model, which comprises the following modules:

[0024] Module M1: obtain structure information of the mixed specialist architecture large language model and hardware feature information of a computing platform, including the number of model layers, the number of specialists in each layer, the number of specialists required to be activated for each word in each layer, the number of parameters of each specialist, the PCIe link bandwidth and the available GPU memory size;

[0025] Module M2: construct a neural network-based specialist access predictor P for each layer of the large language model i The predictor is responsible for predicting the specialists required to be activated in the current layer according to the input before the kth layer, wherein P i is the predictor of the ith layer, which is composed of a double-layer perceptron, the input h i-k is the input of the ith-kth layer of the large language model, the output e i is the probability of each specialist in the ith layer being activated, and k is a prediction distance specified by a user in advance;

[0026] Module M3: collect historical data in the inference process of the large language model as training data, and record the input of each layer gating function and the actually selected specialists for each inference process of each word;

[0027] Module M4: use the collected training data to train the specialist access predictor of each layer offline, and persist to a disk;

[0028] Module M5: start an inference service, load non-specialist parameters into GPU memory, load specialist parameters into CPU memory, and load the corresponding predictor of each layer of the large language model into CPU memory;

[0029] Module M6: during online inference, collect the gating function input of each layer in the GPU inference process into the CPU memory, and after the predictor corresponding to the collected input obtains the input, calculate e i =P i (h i-k ), and output the probability of each specialist being used after the kth layer as a prediction result;

[0030] Module M7: according to the prediction result, select the same number of specialists as the number of activated specialists of the model for each layer, and put them into a prefetch queue, which is executed by a prefetcher to load the specialist parameters from the host memory to the GPU memory according to the order of entering the queue;

[0031] Module M8: when the inference reaches the predicted layer, the specialists not prefetched due to prediction errors are put into the prefetch queue again, which is executed by the prefetcher to perform prefetching;

[0032] Module M9: After the reasoning of one word is completed, the reasoning result is returned to the user, and module M6 is triggered to start the reasoning of the next word.

[0033] Preferably, in module M4, the experts of each layer are trained offline, specifically: for each input in the training data, the output of the predictor is compared with the actual output, the loss function is calculated, and the predictor parameters are updated using backpropagation combined with the gradient descent method, and the above process is repeated for the training data until the predictor converges.

[0034] Preferably, in module M6, the collection process and the prediction process are performed on the CPU in parallel with the GPU reasoning process.

[0035] Preferably, in module M7, the expert parameter loading process is performed in parallel with the GPU reasoning process, and the number of pre-fetched experts is dynamically adjusted according to the accuracy of the predictor.

[0036] Preferably, in module M8, the re-prefetching caused by prediction errors enters the prefetch queue with high priority, and the unfinished error prefetch tasks in the queue are cleared.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] (1) The present application uses active cache updating, so that the large model reasoning process and the expert parameter loading process can be performed in parallel, reducing the parameter loading delay on the critical path and improving the system performance and hardware resource utilization;

[0039] (2) The present application can adjust the prediction strategy by modifying the size of the prediction distance k, achieving an optimal balance between prediction accuracy and prefetching advance, thereby maximizing the prefetching effect;

[0040] (3) The present application accurately predicts the expert access pattern through a neural network predictor. The neural network predictor can learn the expert access pattern from historical information, thereby maintaining a high prediction accuracy at a long prediction distance and effectively improving the loading efficiency of expert parameters;

[0041] (4) The present application has universality and scalability, and can adapt to different model structures, hardware platforms and application scenarios, and improve the generalization ability of the predictor by collecting training data in actual application scenarios;

[0042] (5) The present application uses prefetching priority design, so that the system can quickly switch to the extraction of the correct expert, effectively reducing the performance loss caused by prediction errors. BRIEF DESCRIPTION OF DRAWINGS

[0043] Other features, objects, and advantages of the application will become apparent from the following detailed description of non-limiting embodiments, when read in connection with the following accompanying drawings:

[0044] Figure 1 The application is suitable for a specialist access prediction method for a hybrid specialist architecture large language model.

[0045] Figure 2 An embodiment scenario of the present specification is shown in the figure. DETAILED DESCRIPTION

[0046] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the application, a number of changes and improvements can be made. These are within the scope of the application.

[0047] Example 1

[0048] As Figure 1 , the application provides a specialist access prediction method for a hybrid specialist architecture large language model. This method not only accurately predicts the specialist to be accessed and reasonably arranges the pre-fetching time, but also flexibly adapts to different model structures and hardware platforms. The specific implementation process is as follows:

[0049] Step one, obtain the structural information of the hybrid specialist architecture large language model and the hardware feature information of the computing platform. These information includes but is not limited to the following important aspects: the total number of layers of the model, the number of specialists contained in each layer, the number of specialists that need to be activated for each input word in each layer, the total number of parameters contained in each specialist, and the key information such as the PCIe link bandwidth of the computing platform, the available GPU memory size, etc.

[0050] Step two, construct a neural network-based specialist access predictor P i for each layer of the model. These predictors will predict the specialists that need to be activated in the current layer according to the input information before the kth layer. Among them, P i is the predictor of the ith layer, which consists of a double-layer perceptron, the input h i-k is the input of the i-kth layer of the large language model, and the output e i is the probability of each specialist being activated in the ith layer; k is the prediction distance, whose specific value is specified by the user according to the actual demand in advance.

[0051] Step three, collect the intermediate result data generated by the model during inference as the training data set. Specifically, for each inference process of an input word, record its input to the gate function at each layer, and the final expert actually activated.

[0052] Step four, use the collected training data to train the expert access predictor of each layer offline, and persist to disk. Specifically, for each input in the training data, compare the output of the predictor with the actual output, calculate the loss function, and use backpropagation with gradient descent method to update the predictor parameters. Repeat the above process by traversing the training data until the predictor converges.

[0053] The loss function is a method to measure the gap between the model's predicted output and the actual label. For different tasks, different loss functions may be selected. For example:

[0054] Regression problem: the commonly used loss function is mean squared error (MSE), whose formula is:

[0055]

[0056] Where N is the number of samples, y i is the true value, is the predicted value.

[0057] Classification problem: the commonly used loss function is cross-entropy loss. For binary classification problems, the formula is:

[0058]

[0059] For multi-classification problems, use Softmax combined with cross-entropy loss.

[0060] Once the loss function is calculated, we can use the backpropagation algorithm to calculate the partial derivative of each parameter to the loss function, i.e. the gradient. Then according to these gradient information, through the gradient descent method to adjust the parameters of the model, in order to minimize the loss function. The update rule of gradient descent can be expressed as:

[0061]

[0062] Where θ j represents the jth parameter, and α is the learning rate, which determines the step size of parameter update.

[0063] The steps of backpropagation algorithm are as follows:

[0064] Forward propagation: Pass the input data through the network and calculate the predicted values.

[0065] Compute loss: Calculate the difference between the predicted values and the true values using the loss function mentioned above.

[0066] Backpropagation: Starting from the last layer, calculate the error term (delta) for each node layer by layer, which reflects the contribution of the node's output to the total error.

[0067] Compute gradient: Calculate the gradient of each weight using the error term.

[0068] Update weights: Update the weights in the network according to the calculated gradient and the pre-set learning rate.

[0069] The above process constitutes a complete training iteration (epoch), which usually needs to be repeated multiple times until the loss function converges to a satisfactory level or reaches the pre-set maximum number of iterations.

[0070] Step five, prepare to start the inference service, load non-expert parameters into GPU memory, and load expert parameters into CPU memory. At the same time, load the corresponding predictor for each layer into CPU memory.

[0071] Step six, during online inference, collect the input of the gating function of each layer in the GPU inference process into the CPU memory. The predictor corresponding to the collected input will calculate e i = P i (h i-k ), output the probability of each expert used after k layers, as the prediction result.

[0072] Step seven, according to the prediction result, select the same number of experts as the number of model activation experts for each layer and put them into the prefetch queue for the prefetcher to perform prefetching. The prefetcher loads the expert parameters from the host memory to the GPU memory according to the order of entry.

[0073] Step eight, when the inference reaches the predicted layer, the experts that have not been prefetched due to prediction errors are put back into the prefetch queue and the prefetcher performs prefetching.

[0074] Step nine, after the inference of a word is completed, the inference result is returned to the user and the process returns to step six to start the inference of the next word.

[0075] In step one, the structure information of the mixed expert architecture large language model and the hardware feature information of the computing platform are obtained. The model has 4 layers, each containing 3 experts, and each word needs to activate 2 experts. In this hardware platform, the memory space of the GPU allows caching 2 experts per layer.

[0076] In step two, a neural network based expert predictor is built for each layer. For example, if the user specifies a prediction distance of 2, one of the predictors will predict the experts to be activated in the second layer based on the intermediate results of the first layer.

[0077] In step three, the history data during the model inference is collected as training data. For each inference of a word, the input of each layer’s gating function and the actually selected experts are recorded.

[0078] In step four, the collected training data is used to train the expert predictors of each layer offline. For example, the gating input of the first layer and the expert activation of the second layer are used as the input and output of the predictor that predicts the second layer based on the first layer, respectively, to train the predictor to convergence.

[0079] In step five, the inference service is prepared to start, the non-expert parameters of each layer are loaded into the GPU memory, and the expert parameters, such as the experts E1, E2, E3 of the second layer, are loaded into the CPU memory. At the same time, the predictors, such as the predictor that predicts the second layer based on the first layer, are loaded into the GPU memory. For example, Figure 2 .

[0080] In step six, during online inference, when the inference reaches the gating function of the first layer, the gating function input is collected into the CPU memory. The input will be sent to the predictor that predicts the second layer based on the first layer for prediction, and the prediction result is that E1, E3 will be activated in the second layer.

[0081] In step seven, according to the prediction result, the E1, E3 predicted to be activated in the second layer are put into the prefetch queue. The prefetcher loads the expert parameters of E1, E3 from the host memory into the expert cache in the GPU memory according to the order of entry into the queue.

[0082] In step eight, the inference process and the above loading process are performed synchronously. When the inference reaches the second layer, the actually activated experts are E1, E2, among which E1 has been prefetched into the expert cache in the GPU, and E2 has not been loaded due to prediction error. E2 will be put into the prefetch queue as a high-priority task and be prefetched by the prefetcher in priority.

[0083] In step nine, after E2 is loaded, the large language model will complete the remaining inference calculation, and the inference result will be returned to the user. The system will start the inference of the next word.

[0084] Embodiment 2

[0085] The application further provides a specialist access prediction system applicable to a mixed specialist architecture large language model, which can be realized by performing the process steps of the specialist access prediction method applicable to the mixed specialist architecture large language model, that is, the specialist access prediction method applicable to the mixed specialist architecture large language model can be understood by those skilled in the art as the preferred implementation of the specialist access prediction system applicable to the mixed specialist architecture large language model.

[0086] According to the application, the specialist access prediction system applicable to the mixed specialist architecture large language model comprises: module M1: obtaining structure information of the mixed specialist architecture large language model and hardware feature information of a computing platform, including the number of model layers, the number of specialists in each layer, the number of specialists required to be activated for each word in each layer, the number of parameters of each specialist, the PCIe link bandwidth and the available GPU memory size; module M2: constructing a neural network-based specialist access predictor P for each layer of the large language model i The predictor is responsible for predicting the specialists required to be activated according to the input before the kth layer, wherein P i is the predictor of the ith layer, which is composed of a double-layer perceptron, the input h i-k is the input of the i-kth layer of the large language model, and the output e i is the probability of each specialist of the ith layer being activated, and k is a prediction distance specified by a user in advance; module M3: collecting historical data in the inference process of the large language model as training data, recording the input of each layer of the gating function and the actually selected specialists for each inference process of each word; module M4: using the collected training data to train the specialist access predictor of each layer offline, and persisting to the disk; module M5: starting the inference service, loading the non-specialist parameters into the GPU memory, loading the specialist parameters into the CPU memory, and loading the corresponding predictor of each layer of the large language model into the CPU memory at the same time; module M6: collecting the gating function input of each layer in the GPU inference process to the CPU memory, and calculating e i =P i (h i-k), output the probability of each expert being used after k layers as a prediction result; module M7: according to the prediction result, select the same number of experts as the number of activated experts of the model for each layer, and put them into the prefetch queue for the prefetcher to perform the prefetch operation; the prefetcher loads the expert parameters from the host memory to the GPU memory according to the order of entering the queue; module M8: when the inference reaches the predicted layer, the experts that are not prefetched due to prediction errors are put back into the prefetch queue for the prefetcher to perform the prefetch operation; module M9: after the inference of a word is completed, the inference result is returned to the user, and module M6 is triggered to start the inference of the next word.

[0087] In module M4, the experts of each layer are accessed by the offline trained predictor, specifically: for each input in the training data, the output of the predictor is compared with the actual output, the loss function is calculated, and the parameters of the predictor are updated using back propagation combined with gradient descent method, and the above process is repeated until the predictor converges.

[0088] In module M6, the collection process and the prediction process will be performed on the CPU in parallel with the GPU inference process.

[0089] In module M7, the expert parameter loading process is performed in parallel with the GPU inference process, and the number of prefetched experts is dynamically adjusted according to the accuracy of the predictor.

[0090] In module M8, the re-prefetch caused by prediction errors enters the prefetch queue with high priority, and the unfinished error prefetch tasks in the queue will be cleared.

[0091] Those skilled in the art know that in addition to implementing the system, device and each module thereof provided by the present application in a pure computer readable program code manner, the same program can also be realized in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within hardware components.

[0092] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A method for expert access prediction suitable for mixed expert architecture large language models, characterized in that, Comprising: Step 1: Obtain the structure information of the mixed expert architecture large language model and the hardware feature information of the computing platform, including the number of model layers, the number of experts in each layer, the number of experts needed to be activated for each word in each layer, the number of parameters of each expert, the PCIe link bandwidth and the available GPU memory size; Step 2: build a neural network-based expert access predictor P for each layer of the large language model i , which is responsible for predicting the experts that need to be activated in the current layer according to the input before the k-th layer, where P i is the predictor of the i-th layer, which consists of a two-layer perceptron, the input h i-k is the input of the i-th-k layer of the large language model, the output e i is the probability of each expert being activated in the i-th layer, and k is the prediction distance specified by the user in advance; Step 3: Collect historical data in the inference process of the large language model as training data, and record the input of each layer gating function and the actually selected expert for each word in a reasoning process; Step 4: Use the collected training data to train the expert access predictor of each layer offline, and persist to disk; Step 5: Start the inference service, load the non-expert parameters into the GPU memory, load the expert parameters into the CPU memory, and load the corresponding predictor of each layer of the large language model into the CPU memory; Step 6: When online inference, the gate function input of each layer in the GPU inference process is collected into the CPU memory, and the predictor corresponding to the collected input will calculate e on the CPU after obtaining the input i = P i (h i-k ), the probability of each expert being used after k layers of output, as the prediction result; Step 7: According to the prediction result, select the same number of experts as the model activated expert for each layer, put them into the prefetch queue, and hand them over to the prefetcher to perform prefetching operation. The prefetcher loads the expert parameters from the host memory to the GPU memory according to the order of entering the queue; Step 8: When the inference reaches the predicted layer, the experts that are not prefetched due to prediction errors are put back into the prefetch queue and handed over to the prefetcher for prefetching operation; Step 9: After the inference of a word is completed, the inference result is returned to the user, and the process returns to step 6 to start the inference of the next word.

2. The method of claim 1, wherein the method is applied to a mixed expert architecture large language model. In step 4, the expert access predictor of each layer is trained offline, specifically: for each input in the training data, compare the output of the predictor with the actual output, calculate the loss function, and use back propagation with gradient descent method to update the predictor parameters. Repeat the process of traversing the training data until the predictor converges.

3. The method of claim 1, wherein the method is applied to a mixed expert architecture large language model. In step 6, the collection process and the prediction process will be performed on the CPU in parallel with the GPU inference process.

4. The method of claim 1, wherein the method is applied to a mixed expert architecture large language model. In step 7, the expert parameter loading process is performed in parallel with the GPU inference process, and the number of prefetched experts is dynamically adjusted according to the accuracy of the predictor.

5. The method of claim 1, wherein the method is applied to a mixed expert architecture large language model. In step 8, the re-prefetching caused by prediction errors enters the prefetch queue with high priority, and the unfinished error prefetching tasks in the queue will be cleared.

6. A system for expert access prediction suitable for hybrid expert architecture large language models, the system comprising: Comprising: Module M1: Obtain the structure information of the mixed expert architecture large language model and the hardware feature information of the computing platform, including the number of model layers, the number of experts in each layer, the number of experts needed to be activated for each word in each layer, the number of parameters of each expert, the PCIe link bandwidth and the available GPU memory size; Module M2: build a neural network-based expert access predictor P for each layer of the large language model i The predictor is responsible for predicting the experts that need to be activated in the current layer according to the input before the k layers, where P i is the predictor of the i-th layer, which consists of a two-layer perceptron, the input h i-k is the input of the i-k-th layer of the large language model, and the output e i is the probability of each expert being activated in the i-th layer, and k is the prediction distance specified by the user in advance; Module M3: Collect historical data in the inference process of the large language model as training data, and record the input of each layer gating function and the actually selected expert for each word in a reasoning process; Module M4: Use the collected training data to train the expert access predictor of each layer offline, and persist to disk; Module M5: Start the inference service, load the non-expert parameters into the GPU memory, load the expert parameters into the CPU memory, and load the corresponding predictor of each layer of the large language model into the CPU memory; Module M6: when online inference, the input of the gating function of each layer in the GPU inference process is collected into the CPU memory, and the predictor corresponding to the collected input will calculate e on the CPU after obtaining the input i = P i (h i-k ), the probability of using each expert after k layers of output is output as the prediction result; Module M7: According to the prediction result, select the same number of experts as the model activation experts for each layer, put them into the prefetch queue, and hand them over to the prefetcher for prefetch operation. The prefetcher loads the expert parameters from the host memory to the GPU memory according to the order of entering the queue. Module M8: When the inference reaches the predicted layer, the experts that are not prefetched due to prediction errors are put back into the prefetch queue and handed over to the prefetcher for prefetch operation. Module M9: After the inference of a word is completed, the inference result is returned to the user, and module M6 is triggered to start the inference of the next word.

7. The expert access prediction system for mixed expert architecture large language models of claim 6, wherein, In module M4, the offline training of experts of each layer accesses the predictor, specifically: for each input in the training data, compare the output of the predictor with the actual output, calculate the loss function, and use back propagation with gradient descent method to update the predictor parameters. Repeat the above process by traversing the training data until the predictor converges.

8. The expert access prediction system for mixed expert architecture large language models of claim 6, wherein, In module M6, the collection process and the prediction process will be performed on the CPU in parallel with the GPU inference process. 9.The expert access prediction system for mixed expert architecture large language models of claim 6, wherein, In module M7, the expert parameter loading process is performed in parallel with the GPU inference process, and the number of prefetched experts is dynamically adjusted according to the accuracy of the predictor.

10. The expert access prediction system for mixed expert architecture large language models of claim 6, wherein, In module M8, the re-prefetching caused by prediction errors enters the prefetch queue with high priority, and the unfinished error prefetch tasks in the queue will be cleared.

Images