Hybrid expert model expert prefetching method, system and terminal device based on hybrid prefetching strategy

Abstract

This application relates to the field of model pre-scheduling and memory optimization technology, and provides a hybrid expert model expert prefetching method, system, and terminal device based on a hybrid prefetching strategy. The method includes: obtaining a first expert prefetching candidate set generated based on a first expert prefetching strategy; obtaining a second expert prefetching candidate set generated based on a second expert prefetching strategy; merging the first and second expert prefetching candidate sets to obtain a fused expert prefetching candidate set; and obtaining a target expert prefetching candidate set based on the fused expert prefetching candidate set. This method can balance prefetching coverage and prediction accuracy, effectively improve expert prefetching hit rate, reduce I / O latency and memory contention, and ensure efficient and stable inference of large-scale hybrid expert models in heterogeneous computing environments.

Description

Technical Field

[0001] This application belongs to the field of model pre-scheduling and memory optimization technology, and in particular relates to a hybrid expert model expert pre-fetching method, system and terminal device based on a hybrid pre-fetching strategy. Background Technology

[0002] In inference scenarios for large-scale Mixture of Experts (MoE) models in heterogeneous computing environments, the number of experts in MoE models is so large that it far exceeds the memory capacity of GPUs / NPUs. Therefore, it is necessary to use expert offloading technology to store inactive experts in low-speed storage and schedule them on demand. The prefetching strategy is the key to masking input / output (I / O) latency and ensuring inference performance.

[0003] Existing prefetching techniques are mainly divided into two categories: sequential prefetching and pattern prefetching. Sequential prefetching prefetches all experts in subsequent layers at fixed distances according to the hierarchical order of the expert model, while pattern prefetching predicts based on the discovery of expert access patterns from historical access sequences. Among them, sequential prefetching cannot adapt to the dynamic randomness of MoE gated routing, resulting in problems such as low hit rate, bandwidth waste, and severe memory contention. Pattern prefetching is highly dependent on historical data accumulation, and the prediction confidence drops sharply in cold start or distributed external input scenarios, which can easily cause cache jitter and tail latency spikes. Neither of these two single strategies can balance prefetch coverage and prediction accuracy, which seriously restricts the efficient deployment of MoE models on resource-constrained devices. Summary of the Invention

[0004] This application provides a hybrid expert model expert prefetching method, system, and terminal device based on a hybrid prefetching strategy, which can effectively improve the expert prefetching hit rate, reduce I / O latency and memory contention, and ensure efficient and stable inference of large-scale hybrid expert models in heterogeneous computing environments.

[0005] In a first aspect, embodiments of this application provide a hybrid expert model expert prefetching method based on a hybrid prefetching strategy, including: Obtain the first expert prefetch candidate set generated based on the first expert prefetch strategy; wherein, the first expert prefetch strategy represents the scheduling strategy that traverses subsequent levels and filters unloaded experts according to the hierarchical order of the hybrid expert model; the first expert prefetch candidate set is the set of experts of subsequent levels that need to be prefetched and loaded in advance after the hybrid expert model infers to the current level; Obtain the second expert prefetch candidate set generated based on the second expert prefetch strategy; wherein, the second expert prefetch strategy represents the scheduling strategy for predicting the experts to be accessed based on historical expert access data; the second expert prefetch candidate set is the set of experts of subsequent levels that need to be prefetched and loaded after the hybrid expert model infers to the current layer; The first expert prefetch candidate set and the second expert prefetch candidate set are merged to obtain the expert prefetch fusion candidate set; The target expert prefetch candidate set is obtained based on the expert prefetch fusion candidate set.

[0006] In this embodiment, the first expert prefetching strategy uses a hierarchical traversal approach to filter out unloaded experts in subsequent levels, generating a basic first expert prefetching candidate set to ensure that experts inevitably used in the inference process are covered. The second expert prefetching strategy predicts the experts to be accessed based on historical access data, generating a supplementary second expert prefetching candidate set to capture potential high-frequency experts with access patterns that are difficult to cover with sequential prefetching. The two candidate sets are then merged to obtain an expert prefetching fusion candidate set, and the target expert prefetching candidate set is obtained based on the expert prefetching fusion candidate set. The first expert prefetching strategy can guarantee the basic prefetching coverage of the inference path and avoid the failure of pattern prefetching in cold start or off-distribution scenarios. The second expert prefetching strategy can discover expert access patterns and make up for the shortcomings of sequential prefetching in adapting to the randomness of dynamic routing. The two strategies are integrated and complementary, reducing cache misses and I / O latency with limited GPU memory and bandwidth resources, as well as reducing GPU memory contention and bandwidth waste, significantly improving the inference efficiency and deployment stability of the hybrid expert model.

[0007] In one possible implementation of the first aspect, a first prefetching module corresponds to the first expert prefetching strategy; the first prefetching module is used to obtain the first index and the first prefetching distance corresponding to the current layer after the hybrid expert model infers to the current layer, so as to traverse the subsequent layers according to the first index and the first prefetching distance, filter unloaded experts, and generate a first expert prefetching candidate set; wherein, the first prefetching distance is the prefetching distance obtained by halving the actual prefetching distance.

[0008] In this embodiment, the design of halving the actual prefetch distance can reduce the range of invalid prefetches, reduce bandwidth and memory resource waste, adapt to the dynamic routing characteristics of the MoE model, reduce memory contention caused by redundant prefetching, and ensure the coverage of basic layer prefetching, effectively improving the resource utilization and inference stability of sequential prefetching.

[0009] In one possible implementation of the first aspect, the first expert prefetch module traverses the target layer backward, and the index corresponding to the target layer is the first prefetch distance. The first expert prefetching module is used to traverse all experts in the target layer and create priority prefetching tasks for unloaded experts to obtain the first expert candidate set.

[0010] In this embodiment, by limiting expert traversal to the target layer only within the first prefetch distance range, the coverage of sequential prefetching can be strictly controlled, avoiding meaningless long-distance redundant prefetching and reducing the waste of memory and bandwidth resources. At the same time, creating priority prefetching tasks for experts not loaded within the target layer can achieve orderly scheduling of prefetching tasks, prioritizing the loading of high-priority experts, effectively reducing memory contention and I / O scheduling conflicts. This not only ensures the basic prefetch coverage of subsequent inference layers but also improves the accuracy and execution efficiency of sequential prefetching, alleviating the problem of excessive resource consumption in traditional fixed-range sequential prefetching.

[0011] In one possible implementation of the first aspect, the second prefetching module corresponds to the second expert prefetching strategy. The second prefetching module is used to obtain historical expert access data, confidence threshold and preset data window after the hybrid expert model infers to the current layer, so that the second prefetching module predicts the experts to be accessed based on the historical expert access sequence, confidence threshold and preset data window, so as to generate a second expert prefetching candidate set.

[0012] In this embodiment, the second prefetch module combines historical expert access data, confidence thresholds, and a preset data window to predict the experts to be accessed. This fully explores the expert access patterns of the hybrid expert model and accurately identifies experts with a high probability of being called. Compared with traditional sequential prefetching, it is more adaptable to the randomness of dynamic routing in the model and can effectively improve the prefetch hit rate. At the same time, relying on the confidence threshold to filter low-confidence prediction results avoids the waste of memory and bandwidth caused by invalid prefetching. Combined with the preset data window to control the calculation range, it reduces the computational overhead while ensuring prediction accuracy. This makes up for the shortcomings of poor adaptability and low resource utilization of single sequential prefetching, and provides reliable predictive prefetching support for efficient inference of hybrid expert models.

[0013] In one possible implementation of the first aspect, where the preset data window is no larger than the historical expert access data, The second expert prefetching module is used to obtain the access mode corresponding to the current layer based on historical expert access data and a preset data window, and to calculate the mode frequency of the access mode in the historical expert access data. The second expert prefetching module is also used to calculate the confidence level of the access pattern based on the pattern frequency, and when the confidence level is greater than the confidence level threshold, to predict the experts to be accessed based on the historical expert access sequence data, thereby obtaining the second expert prefetching candidate set.

[0014] In this embodiment, by limiting the preset data window to no larger than the historical expert access data, access patterns, calculation patterns frequency, and corresponding confidence levels are extracted based on the historical data within the window. Expert prediction is only performed when the confidence level exceeds a set threshold. This effectively filters out invalid access patterns with low reliability and low frequency, avoiding erroneous prefetching due to insufficient historical data or low pattern confidence, and significantly improving the accuracy and stability of pattern prefetching. At the same time, relying on historical access sequences to achieve accurate prediction of the experts to be accessed can adapt to the random characteristics of dynamic routing in hybrid expert models, making up for the shortcomings of traditional sequential prefetching in matching access patterns. This reduces the waste of GPU memory and bandwidth caused by invalid prefetching, and also reduces cache jitter and inference tail latency, ensuring efficient and stable inference of large-scale hybrid expert models in heterogeneous computing environments.

[0015] In one possible implementation of the first aspect, the second expert access module is further configured to store the pattern frequency corresponding to the current access mode in a pre-established pattern frequency storage table when the confidence level corresponding to the current access mode is greater than the confidence level threshold. When calculating the pattern frequency of a new access pattern, the second expert access module is also used to detect whether the new access pattern is stored in the pattern frequency storage table. If the new access mode is stored in the mode frequency storage table, the second expert access module is also used to obtain the historical mode frequency corresponding to the new access mode from the mode frequency storage table, and increment the count of the historical mode frequency to obtain the current mode frequency of the new access mode.

[0016] In this embodiment, by establishing a pattern frequency storage table, frequency data of high-confidence valid access patterns is persistently stored and reused. When calculating the frequency of a new access pattern, the historical frequency is directly looked up and reused to complete the count update, eliminating the need to repeatedly traverse the entire historical data for repeated statistics. This significantly reduces the computational overhead and time latency of pattern frequency calculation and improves the processing efficiency of pattern prefetching. At the same time, the frequency data of valid access patterns can be continuously accumulated to continuously optimize the confidence calculation accuracy of access patterns, avoid resource waste caused by repeated calculations, and further improve the stability and real-time performance of pattern prefetching during hybrid expert model inference, better adapting to the low-latency inference requirements of large-scale heterogeneous computing scenarios.

[0017] In one possible implementation of the first aspect, the target expert prefetch candidate set is obtained based on the expert prefetch fusion candidate set, including: The expert prefetch fusion candidate set is deduplicated to obtain the deduplicated expert prefetch fusion candidate set. The experts in the deduplicated expert prefetch fusion candidate set are sorted according to priority to obtain the target expert prefetch candidate set.

[0018] In this embodiment, by first deduplicating the fused expert prefetch candidate set and then sorting it according to a preset priority, the memory occupation, bandwidth waste, and scheduling redundancy caused by the same expert being repeatedly prefetched can be effectively avoided. At the same time, priority sorting enables high-value experts to be prefetched first, and limited computing and storage resources are reasonably allocated, which greatly improves the accuracy and resource utilization of prefetch scheduling. This not only solves the problem of candidate set duplication and redundancy caused by dual-strategy fusion, but also ensures the orderliness of prefetch execution, further reduces inference latency, and improves the inference efficiency and operational stability of large-scale hybrid expert models.

[0019] Secondly, embodiments of this application provide a hybrid expert model expert offloading system based on a hybrid prefetching strategy, including a first expert prefetching module, a second expert prefetching module, and a hybrid scheduling module; The first expert prefetching module is used to generate a first expert prefetching candidate set based on the first expert prefetching strategy; The second expert prefetching module is used to generate a second expert prefetching candidate set based on the first expert prefetching strategy; The hybrid scheduling module is used to obtain the first expert prefetch candidate set and the second expert prefetch candidate set; The hybrid scheduling module is also used to merge the first expert prefetch candidate set and the second expert prefetch candidate set to obtain the expert prefetch fusion candidate set, and to obtain the target expert prefetch candidate set based on the expert prefetch fusion candidate set.

[0020] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the hybrid expert model expert prefetching method based on a hybrid prefetching strategy as described in any of the first aspects above.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the hybrid expert model expert prefetching method based on a hybrid prefetching strategy as described in any of the first aspects above.

[0022] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the hybrid expert model expert prefetching method based on a hybrid prefetching strategy described in any of the first aspects above.

[0023] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the expert prefetching method for a hybrid expert model based on a hybrid prefetching strategy provided in this application embodiment; Figure 2 This is a schematic diagram of the structure for obtaining the first expert prefetch candidate set provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure for obtaining the second expert prefetch candidate set provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the process of obtaining the target expert pre-selection candidate set provided by the implementation of this application; Figure 5 This is a schematic diagram of the expert prefetching system based on a hybrid expert model with a hybrid prefetching strategy provided in this application. Figure 6 This is a schematic diagram of the overall architecture of the hybrid expert model expert prefetching method based on a hybrid prefetching strategy provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0032] In inference scenarios for large-scale (MoE) models in heterogeneous computing environments, the number of experts in MoE models is enormous, far exceeding the memory capacity of GPUs / NPUs. Therefore, it is necessary to use expert offloading technology to store inactive experts in low-speed storage and schedule them on demand. The prefetching strategy is the key to masking I / O latency and ensuring inference performance.

[0033] Existing prefetching techniques are mainly divided into two categories: sequential prefetching and pattern prefetching. Sequential prefetching prefetches all experts in subsequent layers at fixed distances according to the hierarchical order of the expert model, while pattern prefetching predicts based on the discovery of expert access patterns from historical access sequences. Among them, sequential prefetching cannot adapt to the dynamic randomness of MoE gated routing, resulting in problems such as low hit rate, bandwidth waste, and severe memory contention. Pattern prefetching is highly dependent on historical data accumulation, and the prediction confidence drops sharply in cold start or distributed external input scenarios, which can easily cause cache jitter and tail latency spikes. Neither of these two single strategies can balance prefetch coverage and prediction accuracy, which seriously restricts the efficient deployment of MoE models on resource-constrained devices.

[0034] To address the aforementioned technical issues, this application provides a hybrid expert model prefetching method based on a hybrid prefetching strategy. This method determines the prefetching range through a preset hierarchical window, generates a sequential prefetching candidate set based on the model inference level and prefetching distance, and generates a pattern prefetching candidate set by mining access patterns from expert historical access sequences. The two candidate sets are then merged, deduplicated, and prioritized to form a unified prefetching task queue, which is then scheduled and loaded. This approach ensures full prefetching coverage while improving prediction accuracy, effectively reducing I / O latency and memory resource contention, alleviating cache jitter issues in cold start and distributed external input scenarios, and enabling efficient and stable inference of large-scale MoE models on resource-constrained heterogeneous devices.

[0035] See Figure 1 This is a flowchart illustrating the expert prefetching method for a hybrid expert model based on a hybrid prefetching strategy provided in this application embodiment. It is intended as an example and not a limitation. The method may include the following steps: S101, Obtain the first expert prefetch candidate set generated based on the first expert prefetch strategy; wherein, the first expert prefetch strategy represents the scheduling strategy that traverses subsequent levels and filters unloaded experts according to the hierarchical order of the hybrid expert model; the first expert prefetch candidate set is the set of experts of subsequent levels that need to be prefetched and loaded in advance after the hybrid expert model infers to the current level.

[0036] In this embodiment, a dedicated candidate set generated by the first expert prefetching strategy is retrieved and obtained. This first expert prefetching strategy is a sequential prefetching strategy, which strictly follows the hierarchical order of the hybrid expert model, traversing all subsequent levels sequentially to filter out experts not yet loaded into memory. This is a standardized expert scheduling method. The first expert prefetching candidate set specifically refers to the set of all subsequent level experts that need to be prefetched, loaded, and deployed in advance to support subsequent inference operations when the hybrid expert model's inference reaches the current level.

[0037] In one embodiment, the first prefetching module corresponds to the first expert prefetching strategy. The first prefetching module is used to obtain the first index and the first prefetching distance corresponding to the current layer after the hybrid expert model infers to the current layer, so as to traverse the subsequent layers according to the first index and the first prefetching distance, filter unloaded experts, and generate the first expert prefetching candidate set. The first prefetching distance is the prefetching distance obtained by halving the actual prefetching distance.

[0038] In the embodiments of this application, the first prefetch module is a functional unit that implements the first expert prefetch strategy (sequential prefetch), also known as the sequential prefetch module. Its core function is to traverse the corresponding level range and filter unloaded experts based on the current level index and the halved prefetch distance when the MoE hybrid expert model infers to the current level, and finally generate the first expert prefetch candidate set corresponding to sequential prefetch.

[0039] For example, after the model inferences to the current layer, the first prefetch module (i.e., the sequential prefetch module) first obtains the first index of the current layer, and then obtains the first prefetch distance obtained by halving the originally set actual prefetch distance. Starting from the current layer index, it traverses the corresponding range of layers according to the prefetch distance, filters out the unloaded experts, and then generates the first expert prefetch candidate set. For example, if the first index of the current inference layer is 10, the set actual prefetch distance is 10 layers, and the halved first prefetch distance is 5 layers, the module will traverse layers 11 to 15 from layer 10, filter out all unloaded experts in these 5 layers, and integrate these experts to obtain the generated first expert prefetch candidate set.

[0040] The above method, which halves the actual prefetch distance, can reduce the range of invalid prefetches, reduce bandwidth and memory waste, adapt to the dynamic routing characteristics of the MoE model, reduce memory contention caused by redundant prefetching, and ensure the coverage of basic level prefetching, effectively improving the resource utilization and inference stability of sequential prefetching.

[0041] In one embodiment, when the first expert prefetching module traverses the target layer backward and the index corresponding to the target layer is at the first prefetch distance, the first expert prefetching module is used to traverse all experts in the target layer and create priority prefetching tasks for unloaded experts to obtain the first expert candidate set.

[0042] In this embodiment of the application, provided that the target layer index is within the prefetch range, all experts in the layer are traversed, and prefetch tasks with priority are created for experts that have not yet been loaded, thus forming the first expert prefetch candidate set, which provides a task basis for subsequent prefetch scheduling.

[0043] For example, the first expert prefetching module first determines whether the target layer index to be traversed is within the valid range defined by the first prefetching distance. If the condition is met, it traverses all experts in the target layer, identifies experts that are not loaded into memory, creates a prefetching task with corresponding priority for each unloaded expert, and aggregates all priority-bearing prefetching tasks to obtain the first expert candidate set. For instance, if the current first prefetching distance is 5 layers and the target layer index is 12, which is within the valid prefetching range, and this layer contains experts A, B, C, and D, where experts A and C are not loaded, the module creates prefetching tasks with corresponding priorities for these two experts respectively, and integrates the tasks to form the first expert candidate set.

[0044] In the above method, by limiting expert traversal to the target layer within the first prefetch distance range, the coverage of sequential prefetching can be strictly controlled, meaningless long-distance redundant prefetching can be avoided, and the waste of memory and bandwidth resources can be reduced. At the same time, creating priority prefetching tasks for unloaded experts in the target layer can achieve orderly scheduling of prefetching tasks, prioritizing the loading of high-priority experts, effectively reducing memory contention and I / O scheduling conflicts. This not only ensures the basic prefetch coverage of subsequent inference layers, but also improves the accuracy and execution efficiency of sequential prefetching, alleviating the problem of excessive resource consumption in traditional fixed-range sequential prefetching.

[0045] See Figure 2 This is a schematic diagram of the structure for obtaining the first expert prefetch candidate set provided in an embodiment of this application, as shown below. Figure 2 As shown, specifically: Step 1: Process Initialization: After entering the process, initialization is performed first: T← The prefetch task set T is initialized to an empty set and is used to store all prefetch tasks to be executed.

[0046] i←1: The level offset i is initialized to 1, which means that the traversal starts from the next level after the current level.

[0047] Step 2: Determine the traversal range: Perform the following judgment: i≤d? (i.e., determine whether i is less than or equal to d, where d is the preset first prefetch distance): If "No": it means that all levels within the prefetch range have been traversed, the process ends, and the prefetch task set T is returned.

[0048] If the answer is "Yes": Continue with the following steps.

[0049] Step 3: Determine the target layer and priority: Calculate the target layer: target_layer←l+i, where l is the index of the current inference layer, that is, the target layer is the i-th layer after the current layer.

[0050] Calculate the prefetch priority: priority←P_base-i. Based on a fixed P_base, the closer the level (the smaller i is), the higher the priority, ensuring that experts closer to the current level are prefetched first.

[0051] Step 4: Traverse all experts within the target layer: Iterate through each expert e in target_layer: Check expert status: IsLoaded(e)? (that is, determine whether expert e has been loaded into video memory).

[0052] If the result is "Yes (Loaded)": Skip this expert and proceed directly to the next expert's judgment.

[0053] If "No (Not Loaded)": Create a prefetch task for the expert: CreateTask(e,priority,0_seq) and add the task to the set T←T∪{task}.

[0054] Determine if there is another expert at the target level: If "yes": Return to step 4 and continue iterating through the next expert.

[0055] If "No": All experts at this level have finished processing. Execute i←i+1 to enter the processing loop of the next level.

[0056] Step 5: Loop End and Result Return: Repeat steps 2 through 4 until all levels within the prefetch range have been processed (i>d), and finally return a set T containing all prefetch tasks, which is the first expert prefetch candidate set of sequential prefetching.

[0057] S102, obtain the second expert prefetch candidate set generated based on the second expert prefetch strategy; wherein, the second expert prefetch strategy represents the scheduling strategy for predicting the experts to be accessed based on historical expert access data; the second expert prefetch candidate set is the set of experts of subsequent levels that need to be prefetched and loaded in advance after the hybrid expert model infers to the current layer.

[0058] In this embodiment, the second expert prefetching strategy can be called the pattern prefetching strategy. It is a prefetching scheduling method that relies on historical expert access sequences to mine access patterns and predict subsequent experts to be accessed. The generated candidate set, namely the second expert prefetching candidate set, is the set of subsequent level experts that need to be prefetched and loaded in advance when the model infers to the current layer, which complements the sequential prefetching strategy (first expert prefetching strategy).

[0059] Obtain the second expert prefetch candidate set generated by the pattern prefetching strategy. The second expert prefetching strategy (i.e., the pattern prefetching strategy) refers to analyzing historical expert access data accumulated during the operation of the hybrid expert model, mining and identifying recurring access patterns and rules, and then predicting the scheduling method of experts that may be accessed in the subsequent inference process of the model. The second expert prefetch candidate set is the set of experts for subsequent levels that need to be prefetched and loaded in advance, determined based on the prediction results after the hybrid expert model has inferred to the current level.

[0060] In one embodiment, the second prefetching module corresponds to the second expert prefetching strategy. The second prefetching module is used to obtain historical expert access data, confidence thresholds, and preset data windows after the hybrid expert model infers to the current layer, so that the second prefetching module predicts the experts to be accessed based on the historical expert access sequence, confidence thresholds, and preset data windows, thereby generating a second expert prefetching candidate set.

[0061] In this embodiment, the second prefetching module can be called the pattern prefetching module. When the hybrid expert model infers to the current layer, the module will obtain historical expert access data, confidence threshold and preset data window. By extracting the historical access sequence within the window, it will mine the access patterns and calculate the confidence of the experts to be accessed later. Experts whose confidence reaches the threshold will be selected, thereby generating the second expert prefetching candidate set corresponding to the pattern prefetching, and providing prediction results based on access patterns for subsequent prefetching scheduling.

[0062] In the above method, the second prefetching module combines historical expert access data, confidence thresholds, and preset data windows to predict the experts to be accessed. This fully explores the expert access patterns of the hybrid expert model and accurately locates experts with a high probability of being called. Compared with traditional sequential prefetching, it is more adaptable to the randomness of dynamic routing in the model and can effectively improve the prefetching hit rate. At the same time, relying on the confidence threshold to filter low-confidence prediction results avoids the waste of memory and bandwidth caused by invalid prefetching. Combined with the preset data window to control the calculation range, it reduces the computational cost while ensuring prediction accuracy. This makes up for the shortcomings of poor adaptability and low resource utilization of single sequential prefetching and provides reliable predictive prefetching support for efficient inference of hybrid expert models.

[0063] In one embodiment, when the preset data window is not larger than the historical expert access data, the second expert prefetching module is used to obtain the access mode corresponding to the current layer based on the historical expert access data and the preset data window, and calculate the mode frequency of the access mode in the historical expert access data. The second expert prefetching module is also used to calculate the confidence level of the access pattern based on the pattern frequency, and when the confidence level is greater than the confidence level threshold, to predict the experts to be accessed based on the historical expert access sequence data, thereby obtaining the second expert prefetching candidate set.

[0064] In this embodiment, the pattern prefetching module, also known as the second expert prefetching module, extracts the access pattern corresponding to the current layer from historical expert access data within a preset data window, counts the frequency of its occurrence in historical data and converts it into confidence. When the confidence exceeds a set threshold, it predicts the experts to be accessed based on the access pattern and finally generates the second expert prefetching candidate set corresponding to the pattern prefetching.

[0065] For example, firstly, based on the current inference layer, the access sequence within a preset data window is extracted from the historical expert access data to extract the expert access pattern corresponding to the current layer; then, the frequency of the access pattern in the historical data is counted and the frequency value is converted into confidence; then, the confidence is compared with a preset threshold. If the confidence is greater than the threshold, the experts that the model may access next are predicted based on the subsequent association of the access pattern in the historical sequence. After integrating these predicted experts, a second expert prefetch candidate set is generated.

[0066] For example, first obtain the expert access history sequence, such as expert A, expert B, expert C, expert B, expert C, expert D, expert A, expert B, expert C, expert C, expert D, expert A, expert A, expert B, expert C. Set the window size to 3, and extract a segment of length 3 from the end of the history sequence as the current access pattern, which is expert A, expert B, expert C. Count that this access pattern appears a total of 5 times in the entire history sequence, and take the number of occurrences as the pattern frequency. Then take the base confidence level of 0.1 and the frequency gain coefficient of 0.1, and substitute them into the formula to calculate the confidence level as min(1,0.1+0.1×5)=0.6. The result is 0.6, which is higher than the preset confidence level threshold of 0.5. Next, we counted the experts that appeared immediately after each occurrence of this access pattern. We found that expert D appeared the most frequently and had the highest proportion of occurrence. Therefore, expert D, which appeared the most frequently, was predicted as the next expert to be accessed and included in the second expert prefetch candidate set.

[0067] In the above method, by limiting the preset data window to no larger than the historical expert access data, access patterns, calculation patterns frequency and corresponding confidence levels are extracted based on the historical data within the window. Expert prediction is only performed when the confidence level exceeds a set threshold. This effectively filters out invalid access patterns with low reliability and low frequency, avoiding erroneous prefetching due to insufficient historical data or low pattern confidence, and significantly improving the accuracy and stability of pattern prefetching. At the same time, relying on historical access sequences to achieve accurate prediction of the experts to be accessed can adapt to the random characteristics of dynamic routing in hybrid expert models, making up for the shortcomings of traditional sequential prefetching in matching access patterns. This reduces the waste of GPU memory and bandwidth caused by invalid prefetching, and also reduces cache jitter and inference tail latency, ensuring efficient and stable inference of large-scale hybrid expert models in heterogeneous computing environments.

[0068] In one embodiment, the second expert access module is further configured to store the pattern frequency corresponding to the current access pattern in a pre-established pattern frequency storage table when the confidence level corresponding to the current access pattern is greater than the confidence level threshold. When calculating the pattern frequency of a new access pattern, the second expert access module is also used to detect whether the new access pattern is stored in the pattern frequency storage table. If the new access mode is stored in the mode frequency storage table, the second expert access module is also used to obtain the historical mode frequency corresponding to the new access mode from the mode frequency storage table, and increment the count of the historical mode frequency to obtain the current mode frequency of the new access mode.

[0069] In this embodiment of the application, the second expert prefetching module first stores the current access mode and its corresponding mode frequency into the mode frequency storage table when the confidence of the current access mode is greater than the threshold. When calculating the mode frequency of a new access mode, it first checks whether the new mode already exists in the storage table. If it already exists, it directly retrieves the historical mode frequency corresponding to the mode from the table, increments the historical frequency value by 1, and obtains the current mode frequency of the new mode without having to re-traverse all historical access sequences to count the number of times. For example, a pattern frequency storage table is pre-established. The current access pattern is Expert A, Expert B, and Expert C. Its confidence level of 0.6 is greater than the preset threshold of 0.5. The historical pattern frequency of this pattern is 5. Therefore, this pattern and frequency 5 are stored in the storage table. When the pattern frequency of this pattern is calculated again, it is detected that it already exists in the table. The historical frequency 5 is directly read and incremented by 1 to obtain the current latest pattern frequency of 6.

[0070] In the above method, by establishing a pattern frequency storage table, the frequency data of effective access patterns with high confidence is persistently stored and reused. When calculating the frequency of a new access pattern, the historical frequency is directly retrieved from the table for reuse and the count is updated, without the need to repeatedly traverse the entire historical data for repeated statistics. This can significantly reduce the computing power overhead and time delay of pattern frequency calculation, improve the processing efficiency of pattern prefetching; at the same time, it can continuously accumulate the frequency data of effective access patterns, continuously optimize the confidence calculation accuracy of access patterns, avoid resource waste caused by repeated calculations, further improve the stability and real-time performance of pattern prefetching in the inference process of the mixture-of-experts model, and better adapt to the low-latency inference requirements in large-scale heterogeneous computing scenarios.

[0071] See Figure 3 , which is a schematic structural diagram of obtaining the second expert prefetch candidate set provided by an embodiment of this application. As Figure 3 shown, specifically: 1. Start and initialization: The process starts from "Start".

[0072] Initialize the task set T to be an empty set (T ← ), and T is used to store all prefetch tasks.

[0073] 2. Data length check: Judgment condition: Is the historical length |H| < k? (That is, to judge whether the total length of the expert access history is less than the pattern length k).

[0074] Yes (data insufficient): It means that the historical data is too little to extract effective access patterns, and directly jump to "Return T" to end the process, and no prefetch tasks are generated this time.

[0075] No: The data is sufficient, and continue to execute the subsequent steps.

[0076] 3. Extract the current access pattern: Execute P_current ← Extract Pattern:截取 a subsequence of length k from the end of the historical sequence as the current access pattern.

[0077] 4. Obtain the pattern frequency: Execute f ← GetPattern Frequency(P_current): Count the number of occurrences of the current access pattern P_current in the entire historical sequence to obtain the pattern frequency f.

[0078] 5. Calculate the confidence: Execute C←C_base+α*f (or C←min(1,α+β·f), where α is the base confidence level (typically 0.1) and β is the frequency gain coefficient (typically 0.1)) to calculate the prediction confidence level C using the base confidence level C_base and the frequency gain coefficient α, combined with the frequency f.

[0079] 6. Confidence threshold determination: Judgment condition: C < confidence threshold? (i.e., determine whether C is less than the confidence threshold) Yes (Insufficient confidence): The pattern prediction result is unreliable. No prefetch task is generated. Jump to "Return T" and end the process.

[0080] No: Confidence level meets the standard; continue with subsequent steps.

[0081] 7. Update the frequency storage table: Execute Update Frequency Table(P_current): Store the current access mode P_current and its frequency f into the mode frequency storage table for easy reuse and incremental frequency updates, avoiding duplicate full statistics.

[0082] 8. Predict the next expert to be visited: Execute e_pred←Predict Next Expert: Based on the subsequent association patterns of the current access pattern P_current in the historical sequence, predict the next expert e_pred that is most likely to be accessed (usually the expert with the highest subsequent frequency of this pattern).

[0083] 9. Expert loading status check: Judgment condition: IsLoaded(e_pred)? (That is, to determine whether the prediction expert e_pred has been loaded into memory).

[0084] Yes (Loaded): No additional prefetch task is needed. Jump directly to "Return to T" to end the process.

[0085] No (Not Loaded): A prefetch task needs to be created before proceeding with the next steps.

[0086] 10. Create a prefetch task and add it to the collection: Execute CreateTask(e_pred, P_pattern): Create a prefetch task for the unloaded prediction expert e_pred and associate it with the corresponding access pattern information.

[0087] Execute T←T∪{task}: Add the newly created prefetch task to the task set T.

[0088] 11. Process complete: All branches eventually converge to "Return T", the process ends, and the generated prefetch task set T is output.

[0089] S103, merge the first expert prefetch candidate set and the second expert prefetch candidate set to obtain the expert prefetch fusion candidate set.

[0090] In this embodiment of the application, the first expert prefetch candidate set (sequential prefetching, generated based on hierarchical order traversal) and the second expert prefetch candidate set (pattern prefetching, generated based on access pattern prediction) are merged to eliminate duplicates and obtain the expert prefetch fusion candidate set. This retains the advantages of the two prefetching strategies, avoids duplicate prefetching, and improves prefetching efficiency and hit rate.

[0091] For example, if the first expert prefetch candidate set is {expert1, expert2, expert3} and the second expert prefetch candidate set is {expert3, expert4}, the expert prefetch fusion candidate set obtained after direct merging is {expert1, expert2, expert3, expert3, expert4}.

[0092] S104, Obtain the target expert prefetch candidate set based on the expert prefetch fusion candidate set.

[0093] In this embodiment of the application, based on the merged expert prefetch fusion candidate set, after further processing, the target expert prefetch candidate set for actual prefetch scheduling is finally determined.

[0094] In the above methods, the first expert prefetch strategy can ensure the basic prefetch coverage of the inference path and avoid the failure of pattern prefetching in cold start or distributed scenarios. The second expert prefetch strategy can discover the access patterns of experts and make up for the defect that sequential prefetching cannot adapt to the randomness of dynamic routing. The two are integrated and complementary. With limited memory and bandwidth resources, it can reduce cache misses and I / O latency, reduce memory contention and bandwidth waste, and greatly improve the inference efficiency and deployment stability of the hybrid expert model.

[0095] In one embodiment, see Figure 4 This is a flowchart illustrating the process of obtaining a pre-selected candidate set of target experts, as provided in this application. Figure 4 As shown, step S104 includes: S201, Perform expert deduplication on the expert prefetch fusion candidate set to obtain the deduplicated expert prefetch fusion candidate set.

[0096] In this embodiment of the application, by removing duplicate expert entries from the expert prefetch fusion candidate set, it is ensured that each expert appears only once in the candidate set, avoiding repeated prefetching of the same expert resource and improving the efficiency of subsequent prefetch scheduling.

[0097] For example, a deduplication operation is performed on the expert prefetch fusion candidate set. All expert identifiers in the set are traversed, each expert's unique instance is retained, and duplicate entries are removed to obtain the deduplicated expert prefetch fusion candidate set. For example, if the expert prefetch fusion candidate set is {expert A, expert B, expert C, expert C, expert D, expert B}, after deduplication, duplicate experts B and C are retained only once, and the final deduplicated expert prefetch fusion candidate set is {expert A, expert B, expert C, expert D}.

[0098] S202, sort each expert in the deduplicated expert prefetch fusion candidate set according to priority to obtain the target expert prefetch candidate set.

[0099] In this embodiment of the application, after deduplication of the expert prefetch fusion candidate set is completed, all experts in the candidate set are rearranged in order according to the fixed priority set pre-set for each expert. After the sorting is completed, the target expert prefetch candidate set that can be used for actual scheduling is finally formed.

[0100] For example, a fixed priority level is first set for each expert in advance. Then, all experts in the deduplicated expert prefetch fusion candidate set are sorted in descending order according to their pre-set priorities. After sorting, the target expert prefetch candidate set is obtained. For example, the priorities are preset in advance: expert C has the highest priority, expert A is the second highest, expert D is the third highest, and expert B is the lowest. The deduplicated expert prefetch fusion candidate set includes expert A, expert B, expert C, and expert D. After re-sorting according to the pre-set priorities, the final target expert prefetch candidate set is expert C, expert A, expert D, and expert B.

[0101] In the above method, by first deduplicating the fused expert prefetch candidate set and then sorting it according to a preset priority, it can effectively avoid the memory occupation, bandwidth waste and scheduling redundancy caused by the same expert being repeatedly prefetched. At the same time, by prioritizing the sorting, high-value experts are prefetched first, and limited computing and storage resources are rationally allocated, which greatly improves the accuracy and resource utilization of prefetch scheduling. This not only solves the problem of candidate set duplication and redundancy caused by dual-strategy fusion, but also ensures the orderliness of prefetch execution, further reduces inference latency, and improves the inference efficiency and operational stability of large-scale hybrid expert models.

[0102] In one embodiment, see Figure 5 This is a schematic diagram of a hybrid expert model expert prefetching system based on a hybrid prefetching strategy provided in this application, including a first expert prefetching module, a second expert prefetching module, and a hybrid scheduling module; The first expert prefetching module is used to generate a first expert prefetching candidate set based on the first expert prefetching strategy; The second expert prefetching module is used to generate a second expert prefetching candidate set based on the first expert prefetching strategy; The hybrid scheduling module is used to obtain the first expert prefetch candidate set and the second expert prefetch candidate set; The hybrid scheduling module is also used to merge the first expert prefetch candidate set and the second expert prefetch candidate set to obtain the expert prefetch fusion candidate set, and to obtain the target expert prefetch candidate set based on the expert prefetch fusion candidate set.

[0103] In this embodiment, the expert prefetching system based on a hybrid expert model using a hybrid prefetching strategy consists of three core modules: a first expert prefetching module, a second expert prefetching module, and a hybrid scheduling module. Each module has a clear division of labor and works together to complete the expert prefetching task.

[0104] The first expert prefetching module generates a corresponding first expert prefetching candidate set according to a preset first expert prefetching strategy, and the second expert prefetching module generates a second expert prefetching candidate set according to a corresponding second expert prefetching strategy. The hybrid scheduling module first obtains the first expert prefetching candidate set and the second expert prefetching candidate set generated by the two modules respectively, and then directly merges the two candidate sets to obtain the expert prefetching fusion candidate set. After that, the fusion candidate set is processed by deduplicating experts and sorting according to the fixed priority set for each expert in advance, and finally the target expert prefetching candidate set that can be used to perform expert prefetching is obtained.

[0105] See Figure 6 This is a schematic diagram of the overall architecture of the hybrid expert model expert prefetching method based on a hybrid prefetching strategy provided in the embodiments of this application, as shown below. Figure 6 As shown, it includes: Phase 1: Cache prefetching (current layer is executing) This phase is triggered by the inference engine, with two prefetching modules working in parallel to generate prefetching tasks.

[0106] 1) Inference engine triggers prefetching: Trigger sequential prefetching: Send an instruction to the sequential prefetching module (i.e. the first expert prefetching module) to prefetch experts from i+1 to i+2 of the current layer.

[0107] Trigger pattern prefetching: Send an instruction to the pattern prefetching module (i.e. the second expert prefetching module) to perform pattern matching based on the access history sequence H (preset data window length k).

[0108] 2) Sequential prefetch module generates tasks: ① Traverse all experts in the target layer (i+1 to i+2) and calculate priority = P_base + l (l is the layer number).

[0109] ② Query the cache status of these experts (IsLoaded(e), i.e., determine whether expert e has been loaded into memory) from the hybrid scheduler (hybrid scheduling module).

[0110] ③ Submission order prefetch task set T_seq (i.e., the first expert prefetch candidate set, which only contains unloaded experts).

[0111] 3) Pattern prefetching module generates tasks: ① Extract the current access pattern: P_current=Extract Pattern(H,k).

[0112] ② Calculate the pattern frequency f and the confidence level C = C_base + α·f.

[0113] ③ If C ≥ confidence threshold: ④ Update Frequency Table: Update Frequency Table(P_current).

[0114] ⑤ Predict the next expert: e_pred = Predict Next Expert(P_current).

[0115] ⑥ Query the cache status of e_pred from the hybrid scheduler (IsLoaded(e_pred) determines whether the next predicted expert has been loaded into memory).

[0116] ⑦ Submit pattern prefetch task T_pattern (i.e., the second expert prefetch candidate set).

[0117] Phase 2: Task Merging and Scheduling The fusion and sorting of the two prefetch tasks are completed by the hybrid scheduler.

[0118] 1) Merge task sets Compute the set of fusion tasks (i.e., the expert prefetch fusion candidate set): T_hybrid=T_seq∪T_pattern.

[0119] 2) Conflict resolution and priority ranking ① Perform conflict resolution (deduplication) on the fusion task set and prioritize it according to preset rules (such as confidence level and layer number).

[0120] ② Generate the final prefetch task queue (i.e., the expert prefetch fusion candidate set).

[0121] Phase 3: Cache Management and Bandwidth Control The cache manager performs a prefetch task to load the expert parameters that were missed.

[0122] ① Check the cache capacity and real-time bandwidth to determine whether the current cache capacity and bandwidth are sufficient to handle prefetching tasks.

[0123] ② If the expert is not loaded and bandwidth allows, send a parameter loading request to the "underlying storage / network", ③ Receive and write the expert parameter data to the cache and update the cache status: IsLoaded(e)=True.

[0124] If bandwidth is insufficient or the cache is full, the prefetch task will be downgraded or delayed.

[0125] Phase 4: Inference Execution and Log Recording 1) The inference engine executes the next layer. ① Request the expert parameters required for layer i+1 from the "hybrid scheduler".

[0126] ② The hybrid scheduler hits the cache and returns parameter data.

[0127] ③ The inference engine performs calculations for the current layer and prepares to move to the next layer.

[0128] 2) Update access history The expert sequence (e_act,i,t) from this visit is appended to the historical visit log H to provide data for the next round of pattern prefetching.

[0129] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0131] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 7As shown, the terminal device 7 of this embodiment includes: at least one processor 70 ( Figure 7 (Only one is shown in the image) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on at least one processor 70. When the processor 70 executes the computer program 72, it implements the steps in any of the above embodiments of the hybrid expert model expert offloading method based on a hybrid prefetching strategy.

[0132] The terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 The example of terminal device 7 is merely an illustration and does not constitute a limitation on terminal device 7. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0133] The processor 70 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0134] In some embodiments, memory 71 may be an internal storage unit of terminal device 7, such as a hard disk or memory of terminal device 7. In other embodiments, memory 71 may be an external storage device of terminal device 7, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on terminal device 7. Furthermore, memory 71 may include both internal and external storage units of terminal device 7. Memory 71 is used to store operating system, applications, boot loader, data, and other programs, such as program code of computer programs. Memory 71 can also be used to temporarily store data that has been output or will be output. This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0135] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0137] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A hybrid expert model expert prefetching method based on a hybrid prefetching strategy, characterized in that, The method includes: Obtain a first expert prefetch candidate set generated based on a first expert prefetch strategy; wherein, the first expert prefetch strategy represents a scheduling strategy that traverses subsequent levels and filters unloaded experts according to the hierarchical order of the hybrid expert model; the first expert prefetch candidate set is a set of experts from subsequent levels that need to be prefetched and loaded after the hybrid expert model infers to the current level. Obtain a second expert prefetch candidate set generated based on the second expert prefetch strategy; wherein, the second expert prefetch strategy represents the scheduling strategy for predicting the experts to be accessed based on historical expert access data; the second expert prefetch candidate set is the set of experts at subsequent levels that need to be prefetched and loaded after the hybrid expert model infers to the current layer; The first expert prefetch candidate set and the second expert prefetch candidate set are merged to obtain the expert prefetch fusion candidate set; The target expert prefetch candidate set is obtained based on the expert prefetch fusion candidate set; The first expert prefetching strategy corresponds to a first prefetching module. The first prefetching module is used to obtain the first index and the first prefetching distance corresponding to the current layer after the hybrid expert model infers to the current layer, so as to traverse the subsequent layers, filter unloaded experts, and generate the first expert prefetching candidate set according to the first index and the first prefetching distance. The first prefetching distance is the prefetching distance obtained by halving the actual prefetching distance. The second prefetching module corresponds to the second expert prefetching strategy; the second prefetching module is used to obtain historical expert access data, confidence threshold and preset data window after the hybrid expert model infers to the current layer, so that the second prefetching module predicts the experts to be accessed according to the historical expert access sequence, confidence threshold and preset data window, so as to generate the second expert prefetching candidate set; The step of obtaining the target expert prefetch candidate set based on the expert prefetch fusion candidate set includes: The expert prefetch fusion candidate set is subjected to expert deduplication processing to obtain a deduplicated expert prefetch fusion candidate set; The target expert prefetch candidate set is obtained by sorting each expert in the deduplicated expert prefetch fusion candidate set according to priority.

2. The hybrid expert model expert prefetching method based on a hybrid prefetching strategy according to claim 1, wherein, When the first expert prefetch module traverses the target layer backward, and the index corresponding to the target layer is within the first prefetch distance; The first expert prefetching module is used to traverse all experts in the target layer and create priority prefetching tasks for unloaded experts to obtain the first expert candidate set.

3. The hybrid expert model expert prefetching method based on a hybrid prefetching strategy according to claim 2, wherein, If the preset data window is not larger than the historical expert access data... The second expert prefetching module is used to obtain the access mode corresponding to the current layer based on the historical expert access data and the preset data window, and to calculate the mode frequency of the access mode in the historical expert access data; The second expert prefetching module is further configured to calculate the confidence level of the access pattern based on the pattern frequency, and if the confidence level is greater than the confidence level threshold, predict the experts to be accessed based on the historical expert access sequence data to obtain the second expert prefetching candidate set.

4. The hybrid expert model expert prefetching method based on a hybrid prefetching strategy according to claim 3, wherein, The second expert access module is further configured to store the pattern frequency corresponding to the current access mode in a pre-established pattern frequency storage table when the confidence level corresponding to the current access mode is greater than the confidence level threshold. When calculating the pattern frequency of a new access pattern, the second expert access module is also used to detect whether the new access pattern is stored in the pattern frequency storage table. If the new access mode is stored in the mode frequency storage table, the second expert access module is further configured to obtain the historical mode frequency corresponding to the new access mode from the mode frequency storage table, and increment the count of the historical mode frequency to obtain the current mode frequency of the new access mode.

5. A hybrid expert model expert prefetch system based on a hybrid prefetch policy, the system comprising: It includes a first expert prefetch module, a second expert prefetch module, and a hybrid scheduling module; The first expert prefetching module is used to generate a first expert prefetching candidate set based on a first expert prefetching strategy; The second expert prefetching module is used to generate a second expert prefetching candidate set based on the first expert prefetching strategy; The hybrid scheduling module is used to obtain the first expert prefetch candidate set and the second expert prefetch candidate set; The hybrid scheduling module is further configured to merge the first expert prefetch candidate set and the second expert prefetch candidate set to obtain an expert prefetch fusion candidate set, and obtain a target expert prefetch candidate set based on the expert prefetch fusion candidate set.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images