Data processing method, terminal device, and storage medium
By predicting and preloading target experts into the pre-storage area of the processing device in the hybrid expert model, and utilizing the high-speed transmission within the processing device, the problem of expert loading latency is solved, thereby improving the inference speed and user experience of the hybrid expert model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG LAB OF ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY (SZ)
- Filing Date
- 2025-12-08
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, hybrid expert models suffer from slower inference speeds and negatively impact user experience due to the additional latency introduced by loading experts from low-speed storage devices to high-speed computing devices.
By predicting the target expert list for the next input data and preloading it into the pre-storage area of the processing device, combined with high-speed transmission within the processing device, the experts are quickly migrated to the active storage area for calculation, reducing loading time.
Significantly reduce the loading time of hybrid expert models during the inference process, improving overall inference speed and user experience.
Smart Images
Figure CN121255115B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer technology, and in particular relates to a data processing method, terminal device and storage medium. Background Technology
[0002] In large language models, Mixture-of-Experts (MoE) models achieve an efficient balance between inference speed and multi-task adaptability through an ingenious dynamic scheduling mechanism, which can flexibly support diverse scenarios such as dialogue, translation, and summarization.
[0003] In actual inference deployments, due to the limited memory capacity of high-speed computing devices, it is impossible to store all experts in high-speed computing devices. The existing solution is to store most expert parameters in low-speed storage devices, and only when it is necessary to call an expert stored in the low-speed storage device for calculation, the expert is loaded from the low-speed storage device to the high-speed computing device. This loading operation introduces additional latency, which in turn seriously slows down the model inference speed and affects the user experience. Summary of the Invention
[0004] This application provides a data processing method, terminal device, and storage medium, which can solve the problem that in the inference process of hybrid expert models, the existing solution introduces additional latency by loading experts from low-speed storage devices to high-speed computing devices, which in turn seriously slows down the model inference speed and affects the user experience.
[0005] In a first aspect, embodiments of this application provide a data processing method, including:
[0006] Determine the first expert list based on the current input data;
[0007] Predict the next input data based on the current input data and historical input data, and determine the target expert list based on the next input data;
[0008] If the target expert in the target expert list is located in the storage device, the target expert is loaded into the pre-storage area of the processing device;
[0009] The first expert is moved to the activation storage area of the processing device;
[0010] The first expert is invoked to perform calculations on the current input data and output the results.
[0011] In one possible implementation of the first aspect, loading the target expert into the pre-storage area of the processing device includes:
[0012] Determine the storage space of the prepared storage area;
[0013] If the reserve storage area is full, identify the experts that need to be replaced in the reserve storage area.
[0014] The expert to be swapped out from the preparatory storage area is swapped out to the storage device, and the target expert is loaded into the preparatory storage area.
[0015] In one possible implementation of the first aspect, determining the expert to be swapped out in the pre-storage area includes:
[0016] Calculate the value score for each expert in the pre-stored area;
[0017] The N experts with the lowest value scores in the preparatory storage area are selected as the experts to be replaced in the preparatory storage area; where N is the number of the target experts located in the storage device.
[0018] In one possible implementation of the first aspect, migrating the first expert to the activation storage area of the processing device includes:
[0019] If the first expert is located in the preparatory storage area, the first expert will be moved to the active storage area;
[0020] If the first expert is located in the storage device, the first expert is loaded into the active storage area.
[0021] In one possible implementation of the first aspect, the step of migrating the first expert to the active storage area if the first expert is located in the preparatory storage area includes:
[0022] Determine the storage space of the activated storage area;
[0023] If the preparatory storage area is full, calculate the value score of each expert in the active storage area, and swap the M experts with the lowest value scores in the active storage area to the storage device, where M is the number of the first experts located in the preparatory storage area;
[0024] Migrate the M first experts located in the pre-storage area to the activation storage area;
[0025] If the first expert is located in the storage device, loading the first expert into the active storage area includes:
[0026] Determine the storage space of the activated storage area;
[0027] If the preparatory storage area is full, calculate the value score of each expert in the active storage area, and swap the L experts with the lowest value scores in the active storage area to the storage device, where L is the number of the first experts located in the storage device;
[0028] The L first experts located in the storage device are loaded into the active storage area.
[0029] One possible implementation of the first aspect also includes:
[0030] Determine the frequency score, average gating score, and expert parameter volume for each expert;
[0031] The value score for each expert is determined based on the frequency score, average gating score, and expert parameter volume.
[0032] In one possible implementation of the first aspect, determining the value score for each expert based on the frequency score, average gating score, and expert parameter volume includes:
[0033] Substituting the frequency score, average gating score, and expert parameter volume into the first formula yields the value score for each expert.
[0034] The first formula is:
[0035] V = w1*F + w2*G - w3*S;
[0036] Wherein, V represents the expert's value score, w1 is the first weighting parameter, F is the frequency score, w2 is the second weighting parameter, G is the average gating score, w3 is the third weighting parameter, and S is the expert parameter volume.
[0037] In one possible implementation of the first aspect, determining the first expert list based on the current input data includes:
[0038] Perform feature mapping on the current input data to obtain the intermediate representation vector of the current input data;
[0039] The routing weight of each expert is calculated based on the intermediate representation vector; the routing weight is used to quantify the suitability of each expert in processing the current input data;
[0040] Select the K experts with the highest routing weights to obtain the first expert list.
[0041] In a second aspect, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method as described in any one of the first aspects above.
[0042] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data processing method as described in any one of the first aspects above.
[0043] Fourthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the data processing method described in any one of the first aspects.
[0044] The beneficial effects of the embodiments in this application compared with the prior art are:
[0045] The data processing method provided in the first aspect of this application involves the following steps: First, in the current inference task, a first expert list is determined based on the current input data, identifying the first expert used in the current inference task to perform calculations on the current input data. Then, the next input data is predicted based on the current and historical input data, and a target expert list is determined based on the next input data. This allows the target experts in the target expert list to be pre-loaded from the storage device to the pre-storage area of the processing device, saving loading time when calling the target experts in the next inference task. Next, the storage location of the first expert in the first expert list is checked. If the first expert is located in the pre-storage area, it is quickly migrated to the active storage area via high-speed internal transmission within the processing device. Finally, the first expert is called to perform calculations on the current input data, and the result is output. This embodiment can significantly reduce the loading time of the hybrid expert model during the inference process and improve the overall inference speed of the hybrid expert model.
[0046] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application;
[0049] Figure 2 This is a schematic flowchart of another data processing method provided in an embodiment of this application;
[0050] Figure 3This is a schematic flowchart of another data processing method provided in an embodiment of this application;
[0051] Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;
[0052] Figure 5 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation
[0053] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0054] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0055] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0056] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0057] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0058] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0059] In large language models, Mixture-of-Experts (MoE) models achieve a highly efficient balance between inference speed and multi-task adaptability through a sophisticated dynamic scheduling mechanism, flexibly supporting diverse scenarios such as dialogue, translation, and summarization. Specifically, when an input sample enters the Mixture-of-Experts model, the gating network first performs deep analysis based on the sample features (such as task type, semantic domain, and text structure) of the input sample, generating corresponding routing weights for each expert. The higher the routing weight, the stronger the matching degree of the corresponding expert in the processing ability of the current input sample. Subsequently, the gating network selects a small number (usually one or two) of the most relevant experts based on the routing weights, and performs targeted calculations on the input sample by only calling the most relevant experts. Finally, the Mixture-of-Experts model dynamically weights and fuses the outputs of these most relevant experts according to the routing weights to obtain the final output result.
[0060] However, as the scale of hybrid expert models continues to increase, the number of experts and the total number of parameters grow exponentially. Due to the limited storage capacity and extremely high cost of high-speed computing devices, it becomes impractical to reside all expert parameters in high-speed computing devices such as GPUs or HBMs for extended periods. The conventional approach to this problem is to store most expert parameters in low-speed storage devices such as CPU memory or solid-state drives, and only load the expert parameters from the low-speed storage device to the high-speed computing device when a specific expert needs to be called for computation. This process is called "swap-in". However, this process has significant limitations: firstly, the loading operation introduces additional latency (especially when the number of expert parameters is large), severely slowing down model inference; secondly, since the capacity of high-speed computing devices is limited, when a new expert needs to be loaded onto the high-speed computing device but the high-speed computing device has insufficient space, some of the expert parameters already resident in the high-speed computing device must be "swapped out" to the low-speed storage device. This frequent "swap-in" and "swap-out" not only further increases latency but may also waste resources by repeatedly loading the same expert.
[0061] To address the aforementioned technical problems, this application provides a data processing method. First, in the current inference task, a first expert list is determined based on the current input data, identifying the first expert used to calculate the current input data. Then, the next input data is predicted based on the current and historical input data, and a target expert list is determined based on the next input data. This allows the target experts in the target expert list to be pre-loaded from the storage device to the pre-storage area of the processing device, saving loading time when calling the target experts in the next inference task. Next, the storage location of the first expert in the first expert list is checked. If the first expert is located in the pre-storage area, it is quickly migrated to the active storage area via high-speed internal transmission within the processing device. Finally, the first expert is called to calculate the current input data, and the result is output. This embodiment can significantly reduce the loading time of the hybrid expert model during the inference process and improve the overall inference speed of the hybrid expert model.
[0062] In this application, the storage device is a low-speed storage device, and the processing device is a high-speed computing device.
[0063] See Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application. It is intended as an example and not a limitation. The method may include the following steps S101 to S105.
[0064] Step S101: Determine the first expert list based on the current input data.
[0065] In this embodiment, for the current inference task, after the current input data T enters the input queue, the gating network of the hybrid expert model parses the core features of the current input data T (including key information such as semantic domain, task type, and text structure) through a built-in feature extraction module (such as a lightweight Transformer or CNN sub-network). Based on the core features parsed by the feature extraction module, it calculates the corresponding routing weights for all candidate experts in the candidate expert pool preset during the model architecture design phase of the hybrid expert model. After the gating network outputs the routing weights of each candidate expert based on the semantic features and task attributes of the current input data T, the gating network determines the first expert corresponding to the current input data T according to preset rules, thus obtaining the first expert list. The preset rules can be, for example, a Top-K selection rule, which selects the top K candidate experts with the highest routing weights as the first experts in the first expert list. Typically, K is very small, taking the value 1 or 2; or, candidate experts with routing weights exceeding a preset threshold are selected as the first experts in the first expert list.
[0066] Step S102: Predict the next input data based on the current input data and historical input data, and determine the target expert list based on the next input data.
[0067] To improve the inference speed of the hybrid expert model, this embodiment predicts the target experts needed for the next inference task during the processing of the current inference task, determining a list of target experts so that they can be pre-loaded during the processing of the current inference task. Specifically, a cache is set up to store the complete expert list output by the gating network, including the first expert list determined based on the current input data of the current inference task, and all historical expert lists determined based on historical input data from previous inference tasks. By analyzing the input data and the corresponding expert lists in the cache, the sequence pattern between the input data and the expert lists can be captured, thereby predicting the next input data and the corresponding target expert list.
[0068] In a specific example, the analysis objects are historical input data in the cache and its associated historical expert list, as well as the current input data and its associated first expert list. A temporal model (such as LSTM or Transformer temporal module) is used to capture the sequential patterns between the input data and the expert list (such as semantic relationships in the context of a dialogue scenario and the progressive relationship of task flow). Based on the captured sequential patterns, the temporal model predicts the core features that the next input data may possess (such as its semantic domain, specific task type, and core requirements). These features are then integrated to form the next input data. Subsequently, referring to the matching logic of the gating network in step S101, based on the predicted core features of the next input data, the corresponding routing weights are calculated for all candidate experts. The target experts are then determined through preset rules (such as Top-K filtering rules or filtering rules based on preset thresholds), resulting in the target expert list.
[0069] In another specific example, a lightweight next-expert prediction model is trained, which takes the first expert list corresponding to the current input data and the historical expert list corresponding to the historical input data as inputs, and predicts the target expert list corresponding to the next input data.
[0070] Step S103: If the target expert in the target expert list is located in the storage device, load the target expert into the preparation storage area in the processing device.
[0071] In this embodiment, the storage paths of all candidate experts in the candidate expert pool are divided into storage devices and processing devices. The storage paths in the processing devices are further divided into two logical parts: a preparatory storage area and an activation storage area. The preparatory storage area stores the experts that need to be called according to the prediction of the next inference task, i.e., the target experts in the target expert list. The activation storage area stores the experts that need to be called according to the current inference task, i.e., the first expert in the first expert list. Before the current inference task is completed, the activation storage area temporarily stores the experts used in the previous inference task.
[0072] After determining the list of target experts, check the current storage path of each expert in the list. If the expert is currently stored on a storage device, it needs to be preloaded from the storage device to the processing device's pre-loaded storage area to save loading time when calling the expert in the next inference task. If the expert is already stored in the pre-loaded or active storage area, there is no need to adjust its storage path; the storage path can be reused, thus reducing resource consumption caused by repeated loading.
[0073] This embodiment avoids the cross-device loading wait time when temporarily loading from the low-speed storage device to the high-speed computing device by preloading the target expert stored in the low-speed storage device to the pre-load storage area of the high-speed processing device during the execution of the next inference task. At the same time, by utilizing the logical isolation between the pre-load storage area and the active storage area, the execution of the current task and the pre-loading of the target expert for the predicted next inference task can be carried out in parallel, without having to wait for the current task to finish before starting the temporary cross-device loading operation for the next inference task. This can significantly shorten the overall inference latency of the hybrid expert model.
[0074] S104: Move the first expert to the activation storage area of the processing device.
[0075] In a specific example, after determining the list of first experts, the first experts are first retrieved synchronously in the active storage area and the reserve storage area of the processing device. If the first expert is already in the active storage area (for example, the first expert is among the experts used in the previous round of inference task temporarily stored in the active storage area), then there is no need to change the storage location of the first expert, and the existing storage path can be reused directly to reduce the waste of resources caused by repeated loading.
[0076] If the first expert is located in the pre-installed storage area (i.e., the predicted target expert was pre-loaded into the pre-installed storage area in parallel during the previous inference task, and the predicted target expert is the same as the first expert), then the first expert in the pre-installed storage area needs to be migrated to the active storage area. Specifically, the first expert is quickly migrated to the active storage area through high-speed internal data transfer (such as copying within GPU memory). This process utilizes the high-speed data transfer capabilities within the processing device (such as copying within GPU memory), which can significantly reduce the migration time.
[0077] If the first expert is stored in the storage device (i.e., the target expert predicted in the previous inference task is inconsistent with the first expert, and the first expert was not loaded into the preparatory storage area in advance), then the first expert stored in the storage device will be directly loaded into the active storage area.
[0078] S105: Call the first expert to calculate the current input data and output the result.
[0079] In this embodiment, after all the first experts have entered the activation storage area, the processing device immediately calls the first experts to perform calculations on the current input data T (such as feature transformation and semantic reasoning based on the expertise of different first experts), and performs weighted fusion of the output results of each first expert according to the routing weight generated by the gating network, and finally outputs the result corresponding to the current inference task. At this point, the current inference task is completed.
[0080] In this embodiment, expert scheduling is transformed from the passive approach of existing technologies to an active approach. By predicting the access needs of target experts, the target experts stored in the storage device are asynchronously preloaded into the pre-storage area of the processing device while the current inference task is being calculated. When the next inference task needs to call the target expert, it can be quickly migrated directly from the pre-storage area to the active storage area, reducing blocking time. Unlike existing technologies, there is no need to wait a long time to load the target expert from the low-speed storage device to the high-speed computing device. Therefore, this embodiment can significantly reduce the loading time of the hybrid expert model during the inference process and improve the overall inference speed of the hybrid expert model.
[0081] In one possible implementation, step S101 further includes steps S1011 to S1013.
[0082] Step S1011: Perform feature mapping on the current input data to obtain the intermediate representation vector of the current input data.
[0083] In this embodiment, the gated network of the hybrid expert model performs feature mapping on the current input data through a built-in feature extraction module. If the current input data is text data, the feature extraction module captures the deep meaning of the language through contextual semantic vectors; if the current input data is image data, the feature extraction module converts visual information into vectors through visual feature extraction; if the current input data is structured data, the feature extraction module compresses redundant information and retains key indicators through dimensionality transformation and feature fusion, thereby obtaining an intermediate representation vector that retains both the core information of the current input data and the expert's expertise feature value.
[0084] Step S1012: Calculate the routing weight of each expert based on the intermediate representation vector; the routing weight is used to quantify the fit of each expert in processing the current input data.
[0085] In this embodiment, the gating network calculates the routing weight of each expert based on the intermediate representation vector, where a higher routing weight indicates a stronger processing capability for that expert.
[0086] Step S1013: Select the K experts with the largest routing weights to obtain the first expert list.
[0087] In this embodiment, the gating network selects K experts with the largest routing weights according to the Top-K filtering rule to obtain the first expert list, where K is the number of experts selected by the gating network to process the current inference task.
[0088] In one possible implementation, such as Figure 2 As shown, step S103 further includes steps S1031 to S1034.
[0089] Step S1031: Determine the storage space of the preparation storage area.
[0090] Step S1032: If the preparation storage area is full, determine the experts that need to be replaced in the preparation storage area.
[0091] Step S1033: Swap the experts that need to be swapped out from the preparation storage area to the storage device, and swap the target expert into the preparation storage area.
[0092] In this embodiment, the preparation storage area has a fixed storage capacity. Before loading the target expert from the storage device into the preparation storage area, the remaining storage space in the preparation storage area needs to be checked. If the storage space in the preparation storage area is full, the storage space in the preparation storage area needs to be released first, that is, some experts in the preparation storage area are swapped out to the storage device. When the storage space in the preparation storage area can accommodate the target expert, the target expert is then swapped into the preparation storage area. If the remaining storage space in the preparation storage area is sufficient, step S1034 is executed to directly load the target expert into the preparation storage area, ensuring that the target expert quickly enters the preparation state.
[0093] In one possible implementation, step S1032 further includes steps S10321 and S10322.
[0094] Step S10321: Calculate the value score of each expert in the preparatory storage area.
[0095] Step S10322: Select the N experts with the lowest value scores in the preparatory storage area as the experts to be replaced in the preparatory storage area; where N is the number of target experts located in the storage device.
[0096] In this embodiment, before swapping some experts from the preparatory storage area to the storage device, it is necessary to determine the experts in the preparatory storage area that need to be swapped out. For example, the value scores of all experts currently stored in the preparatory storage area are calculated according to a preset calculation formula. Experts with higher value scores play a greater role in subsequent reasoning tasks. Therefore, after obtaining the value scores of all experts in the preparatory storage area, the value scores are sorted, and the N experts with the lowest value scores in the preparatory storage area are selected as the experts to be swapped out. Then, the N experts with the lowest value scores are swapped to the storage device. Finally, the N target experts located in the storage device are loaded into the preparatory storage area. After the N target experts are loaded into the preparatory storage area, the value scores of all experts in the preparatory storage area are recalculated according to the preset calculation formula. At this time, the value scores of the target experts are higher than before.
[0097] In a specific example, if there are 4 target experts on the storage device and the remaining storage space in the preparation storage area is sufficient for 5 experts, then the remaining storage space in the preparation storage area is sufficient, and the target experts are directly loaded into the preparation storage area. If there are 4 target experts on the storage device and the remaining storage space in the preparation storage area is full, then the value scores of all experts in the preparation storage area are calculated, the 4 experts with the lowest value scores in the preparation storage area are swapped out to the storage device, and then the 4 target experts located on the storage device are loaded into the preparation storage area. If there are 4 target experts on the storage device and the remaining storage space in the preparation storage area is sufficient for 2 experts, then the value scores of all experts in the preparation storage area are calculated, the 2 experts with the lowest value scores in the preparation storage area are swapped out to the storage device, and then the 4 target experts located on the storage device are loaded into the preparation storage area.
[0098] In this embodiment, the reserve storage area can achieve efficient turnover of storage resources under the constraint of fixed capacity by first replacing low-value scoring experts and then replacing them with target experts through dynamic replacement operation. This avoids the problem that the target experts cannot be loaded in advance when the reserve storage area is insufficient.
[0099] In one possible implementation, step S104 further includes steps S1041 and S1042.
[0100] Step S1041: If the first expert is located in the preparatory storage area, move the first expert to the active storage area.
[0101] In this embodiment, if the first expert is located in the pre-storage area (i.e., the predicted target expert is pre-loaded into the pre-storage area in parallel during the previous inference task, and the predicted target expert is the same as the first expert), then the first expert in the pre-storage area is quickly migrated to the activation storage area.
[0102] Step S1042: If the first expert is located in the storage device, load the first expert into the active storage area.
[0103] In this embodiment, if the first expert is stored in the storage device (i.e., the target expert predicted in the previous inference task is inconsistent with the first expert, and the first expert has not been loaded into the preparatory storage area in advance), then the first expert stored in the storage device will be directly loaded into the active storage area.
[0104] In one possible implementation, such as Figure 3 As shown, step S1041 further includes steps S10411 to S10414.
[0105] Step S10411: Determine the storage space of the active storage area.
[0106] Step S10412: If the preparation storage area is full, calculate the value score of each expert in the activation storage area, and swap the M experts with the lowest value scores in the activation storage area to the storage device, where M is the number of the first experts located in the preparation storage area.
[0107] Step S10413: Migrate the M first experts located in the preparatory storage area to the active storage area.
[0108] In this embodiment, if the first expert is stored in the pre-storage area (i.e., the predicted target expert is pre-loaded into the pre-storage area in parallel during the previous inference task, and the predicted target expert is consistent with the first expert), then the first expert is quickly migrated to the activation storage area through high-speed internal transfer (such as copying within GPU memory) of the processing device. This process utilizes the high-speed data transfer capability (such as copying within GPU memory) of the processing device, which can significantly shorten the migration time.
[0109] In this embodiment, the active storage area also has a fixed storage capacity. Before migrating the first expert from the preparatory storage area to the active storage area, the remaining storage space in the active storage area needs to be checked. If the active storage area is full, its storage space needs to be released first, that is, some experts in the active storage area are swapped out to the storage device. Once the active storage area has enough space to accommodate the first expert, the first expert is quickly migrated to the active storage area. If the active storage area has sufficient remaining storage space, step S10414 is executed to directly migrate the first expert to the active storage area, ensuring that the first expert can quickly perform calculations.
[0110] In this embodiment, if the storage space of the active storage area is full, before swapping some experts from the active storage area to the storage device, it is necessary to determine the experts that need to be swapped out from the active storage area. For example, according to a preset calculation formula, the value scores of all experts currently stored in the active storage area are calculated. Experts with higher value scores play a greater role in subsequent reasoning tasks. Therefore, after obtaining the value scores of all experts in the active storage area, the value scores are sorted, and the M experts with the lowest value scores in the active storage area are selected as the experts that need to be swapped out. Then, the M experts with the lowest value scores are swapped to the storage device. Finally, the M first experts located in the reserve storage area are quickly migrated to the active storage area. After the M first experts are migrated to the active storage area, the value scores of all experts in the active storage area are recalculated according to the preset calculation formula. At this time, the value scores of the first experts are higher than before.
[0111] In this embodiment, the active storage area can achieve efficient turnover of storage resources under the constraint of the fixed capacity of the active storage area by first replacing the low-value scoring experts and then replacing them with the first expert through a dynamic replacement operation. This avoids the problem that the first expert cannot be migrated when the active storage area is short of storage space.
[0112] In one possible implementation, such as Figure 3 As shown, step S1042 further includes steps S10421 to S10424.
[0113] Step S10421: Determine the storage space of the active storage area.
[0114] Step S10422: If the preparation storage area is full, calculate the value score of each expert in the active storage area, and swap the L experts with the lowest value scores in the active storage area to the storage device, where L is the number of the first experts located in the storage device.
[0115] Step S10423: Load the L first experts located on the storage device into the active storage area.
[0116] In this embodiment, before loading the first experts stored in the storage device into the active storage area, the remaining storage space in the active storage area needs to be checked. If the storage space in the active storage area is full, the storage space in the active storage area needs to be released first, that is, L low-value experts in the active storage area are swapped out to the storage device. After the storage space in the active storage area can accommodate L first experts located in the storage device, the L first experts located in the storage device are directly loaded into the active storage area. After the L first experts are loaded into the active storage area, the value scores of all experts in the active storage area are recalculated according to a preset calculation formula. At this time, the value scores of the first experts are higher than before. If the remaining storage space in the active storage area is sufficient, step S10424 is executed to directly load the experts stored in the storage device into the active storage area, ensuring that the first experts can perform calculations.
[0117] It should be noted that since the target expert predicted by the previous inference task may be inconsistent with the current first expert, this embodiment still has the scenario of directly loading the first expert stored in the storage device into the active storage area. However, after continuous optimization of the prediction task, most target experts are consistent with the first expert. This embodiment significantly reduces the long latency caused by the existing solution of directly loading experts from low-speed storage devices to high-speed computing devices, improves the overall inference speed of the hybrid expert model, and enhances the user experience.
[0118] In one possible implementation, the method further includes:
[0119] Step S106: Determine the frequency score, average gating score, and expert parameter volume for each expert.
[0120] Step S107: Determine the value score for each expert based on the frequency score, average gating score, and expert parameter volume.
[0121] In one possible implementation, step S107 further includes:
[0122] Substituting the frequency score, average gating score, and expert parameter volume into the first formula yields the value score for each expert; whereby the first formula is:
[0123] V = w1*F + w2*G - w3*S;
[0124] Wherein, V represents the expert's value score, w1 is the first weighting parameter, F is the frequency score, w2 is the second weighting parameter, G is the average gating score, w3 is the third weighting parameter, and S is the expert parameter volume.
[0125] In this embodiment, the value score of each expert not only takes into account the time when the expert was last visited, but also integrates multiple optimization objectives such as the frequency score of visiting experts, the average gating score, and the expert parameter volume. Based on the value score of the expert calculated by the first formula in this embodiment, the experts to be swapped out in the reserve storage area and the active storage area are determined. This can better conform to the access characteristics of the hybrid expert model, reduce expert turbulence, and make the overall value of the experts residing in the processing device higher.
[0126] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0127] Corresponding to the data processing method described in the above embodiments, this application provides a structural block diagram of a data processing device. For ease of explanation, only the parts related to the embodiments of this application are shown.
[0128] Reference Figure 4 The data processing device includes:
[0129] The first determining module 41 is used to determine the first expert list based on the current input data;
[0130] The second determining module 42 is used to predict the next input data based on the current input data and historical input data, and to determine the target expert list based on the next input data;
[0131] Loading module 43 is used to load the target experts in the target expert list into the pre-storage area of the processing device if the target experts in the target expert list are stored in the storage device.
[0132] Migration module 44 is used to migrate the first expert to the activation storage area of the processing device;
[0133] Output module 45 is used to call the first expert to calculate the current input data and output the result.
[0134] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0135] in addition, Figure 4 The data processing device shown can be a software unit, hardware unit, or a combination of software and hardware built into an existing terminal device, or it can be integrated into the terminal device as an independent component, or it can exist as an independent terminal device.
[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0137] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 5 As shown, the terminal device 5 in this embodiment includes: at least one processor 50 ( Figure 5 The diagram shows only one processor, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 executes the computer program 52 to implement the steps in any of the above-described data processing method embodiments.
[0138] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 This is merely an example of terminal device 5 and does not constitute a limitation on terminal device 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.
[0139] The processor 50 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0140] In some embodiments, the memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. In other embodiments, the memory 51 may be an external storage device of the terminal device 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 5. Furthermore, the memory 51 may include both internal and external storage units of the terminal device 5. The memory 51 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 51 can also be used to temporarily store data that has been output or will be output.
[0141] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.
[0142] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.
[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0144] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0146] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0148] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A data processing method, characterized in that, include: Determine the first expert list based on the current input data; Predict the next input data based on the current input data and historical input data, and determine the target expert list based on the next input data; If the target expert in the target expert list is located in the storage device, the target expert is loaded into the pre-storage area of the processing device; wherein, the pre-storage area is used to store the experts that need to be called according to the prediction of the next inference task; The first expert is migrated to the activation storage area of the processing device; wherein, the activation storage area is used to store the experts that need to be invoked according to the current inference task; The first expert is invoked to calculate the current input data and output the result; The step of loading the target expert into the preparatory storage area of the processing device includes: determining the storage space of the preparatory storage area; if the preparatory storage area is full, calculating the value score of each expert in the preparatory storage area; selecting the N experts with the lowest value scores in the preparatory storage area as the experts to be replaced from the preparatory storage area, where N is the number of the target experts located in the storage device; replacing the experts to be replaced from the preparatory storage area into the storage device, and loading the target expert into the preparatory storage area; The method further includes: determining the frequency score, average gating score, and expert parameter size for each expert; substituting the frequency score, average gating score, and expert parameter size into a first formula to obtain the value score for each expert; the first formula is: V = w1*F + w2*G - w3*S; Wherein, V represents the expert's value score, w1 is the first weighting parameter, F is the frequency score, w2 is the second weighting parameter, G is the average gating score, w3 is the third weighting parameter, and S is the expert parameter volume.
2. The data processing method according to claim 1, characterized in that, The step of migrating the first expert to the activation storage area of the processing device includes: If the first expert is located in the preparatory storage area, the first expert will be moved to the active storage area; If the first expert is located in the storage device, the first expert is loaded into the active storage area.
3. The data processing method according to claim 2, characterized in that, If the first expert is located in the preparatory storage area, migrating the first expert to the active storage area includes: Determine the storage space of the activated storage area; If the active storage area is full, calculate the value score of each expert in the active storage area, and swap the M experts with the lowest value scores in the active storage area to the storage device, where M is the number of the first experts located in the reserve storage area; Migrate the M first experts located in the pre-storage area to the activation storage area; If the first expert is located in the storage device, loading the first expert into the active storage area includes: Determine the storage space of the activated storage area; If the active storage area is at full capacity, calculate the value score of each expert in the active storage area, and swap the L experts with the lowest value scores in the active storage area to the storage device, where L is the number of the first experts located in the storage device; The L first experts located in the storage device are loaded into the active storage area.
4. The data processing method according to claim 1, characterized in that, The step of determining the first expert list based on the current input data includes: Perform feature mapping on the current input data to obtain the intermediate representation vector of the current input data; The routing weight of each expert is calculated based on the intermediate representation vector; the routing weight is used to quantify the suitability of each expert in processing the current input data; Select the K experts with the highest routing weights to obtain the first expert list.
5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.
6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.