Text data reasoning method and device based on hybrid expert model

Abstract

The invention discloses a text data reasoning method and device based on a hybrid expert model, which are used for solving the technical problem of low reasoning efficiency caused by the existing reasoning process based on the hybrid expert model. The method comprises the steps of obtaining a pre-running text data set; inputting the pre-running text data set into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-running reasoning, and determining hybrid expert module calculation transmission time and an expert correlation table; based on the expert correlation table and a preset number of prefetched hot experts, performing hot expert identification on a preset hybrid expert model, and determining a plurality of target hybrid expert modules; adopting a preset constraint perception planning inequality to calculate transmission time according to a preset number of prefetched hot experts and a hybrid expert module, and determining the batch number of input texts; and in the multi-level heterogeneous memory space, outputting a model reasoning result by adopting each target hybrid expert module according to the to-be-reasoned text data subset corresponding to the batch number of the input texts.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text data reasoning method and device based on a hybrid expert model. Background Art

[0002] In recent years, with the rapid development of deep learning, large models have achieved remarkable results in various fields. In order to scale the model without increasing the training and inference costs, recent work has introduced the sparsely activated mixture of experts model, namely Mixture-of-Experts (MoE). The MoE model usually uses the MoE layer to replace the feed-forward network (FFN) layer. For each input, it only requires sparsely activated parameters (i.e., experts) to participate in the calculation, which greatly reduces the computational cost.

[0003] The current work has proved the superiority of the MoE structure through a large number of experiments, but the MoE model faces more severe challenges in the graphics memory bottleneck during the inference stage than other large models, which is mainly due to the large parameter size of the MoE. For example, DeepSeek-V2 has 236B parameters, which requires at least eight of the most advanced GPUs (such as A100) to perform inference. This challenge is particularly serious when resources are limited.

[0004] Most of the existing reasoning processes based on hybrid expert models directly apply traditional dense models to MoE models, pre-fetching the next layer when performing calculations on the current layer. They do not consider the differences in IO resource requirements of different layers, and directly pre-fetch the entire MoE layer, resulting in a large number of "bubbles" in the pipeline, leading to low reasoning efficiency. Summary of the invention

[0005] The present invention provides a text data reasoning method and device based on a hybrid expert model, which are used to solve the technical problem that the existing reasoning process based on the hybrid expert model leads to low reasoning efficiency.

[0006] The first aspect of the present invention provides a text data reasoning method based on a hybrid expert model, comprising:

[0007] Respond to inference requests, build multi-level heterogeneous memory spaces, and obtain pre-run text datasets;

[0008] Inputting the pre-run text data set into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, determining a hybrid expert module calculation transmission time and an expert correlation table;

[0009] Based on the expert correlation table and the preset number of pre-fetched hot experts, performing hot expert identification on multiple hybrid expert modules in the preset hybrid expert model to determine multiple target hybrid expert modules;

[0010] The preset constraint-aware planning inequality is used to calculate the transmission time according to the preset number of pre-fetched popular experts and the hybrid expert module to determine the number of input text batches;

[0011] In the multi-level heterogeneous memory space, each target hybrid expert module is used to output a model reasoning result corresponding to the subset of text data to be inferred according to the subset of text data to be inferred corresponding to the number of input text batches.

[0012] Optionally, based on the expert correlation table and the preset number of pre-fetched hot experts, performing hot expert identification on multiple hybrid expert modules in the preset hybrid expert model to determine multiple target hybrid expert modules includes:

[0013] Based on the expert relevance table, determining the expert selection frequencies of the plurality of pre-operation text data in the pre-operation text data set for the plurality of experts in each of the hybrid expert modules;

[0014] Sorting the expert selection frequencies of each of the pre-running text data for each of the experts in each of the hybrid expert modules in descending order;

[0015] Among the experts in each of the mixed expert modules, experts corresponding to the expert selection frequencies of the pre-set number of popular experts are selected as popular experts, and the remaining experts are selected as non-popular experts;

[0016] The mixed expert module corresponding to each of the popular experts is used as the target mixed expert module.

[0017] Optionally, the target hybrid expert module includes a gating layer, an expert layer and an attention layer; the expert layer includes a hot expert with a preset number of hot experts and a plurality of non-hot experts; the data processing process of the target hybrid expert module is specifically as follows:

[0018] Performing weight calculation on a plurality of input text unit data in the input text data subset input to the target hybrid expert module through a gating layer, and determining a weight value corresponding to each non-hot expert for each input text unit data;

[0019] The weight values ​​corresponding to any input text unit data of each of the non-hot experts are sorted in descending order, and non-hot experts corresponding to a preset number of weight values ​​are selected as non-hot activated experts;

[0020] Performing statistics on the non-hot activation experts corresponding to each of the input text unit data to determine the activation frequency corresponding to each of the non-hot activation experts;

[0021] Sorting the activation frequencies corresponding to the non-hot activation experts in descending order to generate a non-hot activation expert data processing and sorting result;

[0022] Use a preset number of hot experts to calculate the input text data subset respectively, and output the hot expert calculation results corresponding to the preset number of hot experts;

[0023] Using the input text data subsets in sequence as inputs of non-hot activation experts corresponding to the non-hot activation expert data processing and sorting results, and outputting a plurality of non-hot activation expert calculation results;

[0024] Merging the hot expert calculation results and the non-hot activated expert calculation results to generate a merged expert calculation result;

[0025] The combined expert calculation results are calculated by the attention layer using a preset attention mechanism to generate an output model result.

[0026] Optionally, the multi-level heterogeneous memory space includes GPU video memory, CPU memory and hard disk; the GPU video memory is used to store tensors required for current calculations and pre-fetched tensors; the CPU memory and the hard disk are used to store inactivated tensors.

[0027] Optionally, different types of tensors are placed in the GPU video memory, the CPU memory and the hard disk respectively according to preset layer-by-layer allocation conditions.

[0028] Optionally, the hybrid expert module calculates the transmission time including the attention layer calculation time, the gating layer calculation time, the popular expert calculation time, the single expert average calculation time, the attention layer transmission time, the gating layer transmission time, and the single expert transmission time; the preset constraint-aware planning inequality is specifically:

[0029] ;

[0030] Where n is the number of input text batches, and n is the smallest integer that satisfies the preset constraint-aware planning inequality; Calculate time for attention layer; Transmission time for the gating layer; Calculate time for the gating layer; Transmit time for a single expert; is the calculation time of popular experts; K is the number of preset and pre-fetched popular experts; is the average computation time of a single expert for the i-th expert; is a built-in function; Q is the number of experts; Transfer time for the attention layer.

[0031] A second aspect of the present invention provides a text data reasoning device based on a hybrid expert model, comprising:

[0032] The response module is used to respond to inference requests, build multi-level heterogeneous memory spaces, and obtain pre-run text datasets;

[0033] A pre-run reasoning module, used for inputting the pre-run text data set into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, and determining a hybrid expert module calculation transmission time and an expert correlation table;

[0034] An identification module, used for performing hot expert identification on a plurality of hybrid expert modules in the preset hybrid expert model based on the expert correlation table and the preset number of hot experts, and determining a plurality of target hybrid expert modules;

[0035] A batch quantity determination module is used to determine the number of input text batches by using a preset constraint-aware planning inequality according to the preset number of pre-fetched popular experts and the transmission time calculated by the hybrid expert module;

[0036] The output result module is used to output the model reasoning result corresponding to the text data subset to be inferred according to the text data subset to be inferred corresponding to the number of input text batches in the multi-level heterogeneous memory space by using each of the target hybrid expert modules.

[0037] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the text data reasoning method based on the hybrid expert model as described in any one of the above items.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the steps of the text data reasoning method based on the hybrid expert model as described in any one of the above items are implemented.

[0039] A fifth aspect of the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer executes the steps of the text data reasoning method based on the hybrid expert model as described in any one of the above items.

[0040] It can be seen from the above technical solutions that the present invention has the following advantages:

[0041] The above technical solution of the present invention provides a text data reasoning method based on a hybrid expert model. When it is necessary to reason about text data, a multi-level heterogeneous memory space is first constructed, and a pre-run text data set is obtained; then, the pre-run text data set is input into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, and the hybrid expert module calculates the transmission time and the expert correlation table to determine; based on the expert correlation table and the preset number of pre-fetched hot experts, hot expert identification is performed on multiple hybrid expert modules in the preset hybrid expert model to determine multiple target hybrid expert modules; a preset constraint-aware planning inequality is used to calculate the transmission time according to the preset number of pre-fetched hot experts and the hybrid expert module to determine the number of input text batches; finally, in the multi-level heterogeneous memory In the space, each target hybrid expert module is used to output the model reasoning result corresponding to the text data subset to be inferred according to the input text batch number. Based on the above scheme, according to the pre-running reasoning process, the hardware capabilities in the current environment are adaptively measured, the hybrid expert module calculates the transmission time and the expert correlation table, and combines the preset constraint perception planning inequality to output the input text batch number, and the text data subset to be inferred corresponding to the input text batch number is used as the input of the multiple target hybrid expert modules determined based on the expert correlation table for popular expert identification, and the model reasoning result is output. The present invention focuses on the currently popular hybrid expert model, and greatly compresses the "bubbles" existing in the reasoning pipeline, thereby improving the reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A flowchart of a method for reasoning text data based on a hybrid expert model provided in the first embodiment of the present invention;

[0044] Figure 2 A schematic diagram of an IO-computing pipeline provided in Embodiment 1 of the present invention;

[0045] Figure 3 A schematic diagram of the arrangement of the computational graph perceived by popular experts provided in the first embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a computational graph traversal provided in the first embodiment of the present invention;

[0047] Figure 5 A flow chart of a text data reasoning method based on a hybrid expert model provided in the second embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of an existing large model provided in the second embodiment of the present invention;

[0049] Figure 7 This is a structural block diagram of a text data reasoning device based on a hybrid expert model provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0050] The embodiment of the present invention provides a text data reasoning method and device based on a hybrid expert model, which are used to solve the technical problem of low reasoning efficiency caused by the existing reasoning process based on the hybrid expert model.

[0051] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Terminology explanation:

[0053] Mixture-of-Experts (MoE): The Mixture-of-Experts model replaces the feedforward neural network layer in the Transformer with the MoE layer. The MoE layer consists of a gating network and multiple experts. The gating function is the key feature of the MoE, which discretely decides the expert selection for each token. The most common practice is to select the top k experts for each token. In this case, the gating function is usually a softmax function, which activates the top-k experts to participate in the calculation according to the calculated probability distribution, and k is generally 1 or 2. Experts are essentially feedforward neural networks (FFN).

[0054] Popular experts: In each MoE layer, for each input token, the gating network selects top-k experts from n experts to process the token. When the number of input tokens is large, some experts are selected and used by the gating mechanism more frequently than other experts, and these experts are called popular experts.

[0055] Token: In natural language processing, a token usually refers to a meaningful piece of text. When processing text, large models will split the input sentence into tokens.

[0056] Throughput: number of tokens generated / time consumed

[0057] Pipeline: The pipeline in a computer breaks down a task into multiple independent stages, each of which can process different tasks in parallel. The IO-computing pipeline below mainly describes the processing order and association of calculation and data transmission in parallel.

[0058] See also Figure 1 , Figure 1 A flowchart of the steps of a text data reasoning method based on a hybrid expert model provided in Example 1 of the present invention.

[0059] The present invention provides a text data reasoning method based on a hybrid expert model, comprising:

[0060] Step 101: respond to inference requests, build a multi-level heterogeneous memory space, and obtain a pre-run text dataset.

[0061] The multi-level heterogeneous memory space includes GPU memory, CPU memory, and hard disk; GPU memory is used to store tensors required for current calculations and pre-fetched tensors; CPU memory and hard disk are used to store inactive tensors. According to the preset layer allocation conditions, different types of tensors are placed in GPU memory, CPU memory, and hard disk respectively.

[0062] It should be noted that the multi-level heterogeneous memory space constructed by the present invention includes GPU memory, CPU memory and hard disk to meet the storage requirements of large models in resource-constrained environments. Through adaptive tensor deployment, various tensors are reasonably placed according to the storage resources in the current environment to improve the utilization of existing resources. Specifically, the GPU memory is mainly used to store necessary parts, namely the tensors required for the current calculation and the pre-fetched tensors. When there are more idle memory resources in the current environment, they can also be further utilized to reduce some IO. Specifically, the present invention can select placement locations for different types of tensors such as experts, gating, attention layers, KV caches, activation values, etc. according to preset layer-by-layer allocation conditions, and supports layer-by-layer placement. For example, the preset layer-by-layer allocation conditions can be set to place the first 3 layers of experts in GPU memory, then 20 layers of experts in CPU memory, and the rest on disk.

[0063] Furthermore, for the inactivated part, you can choose to unload it to the CPU memory or hard disk. The present invention gives experts a higher priority to occupy the CPU memory. Because the challenge faced by the MoE layer is that the experts requested each time are determined by the gating function and cannot be accurately predicted in advance. Therefore, in the face of large-scale tasks, there is a high probability that experts will need to be transmitted in real time, so it is urgent to quickly transfer the required expert tensors to the video memory. Considering that CPU memory with a faster transmission rate can provide a faster response speed, the expert part is given priority to occupy the CPU memory to deal with unexpected situations.

[0064] It is worth mentioning that when the CPU memory is sufficient, the present invention uses pin_memory to achieve faster CPU-GPU transmission. When the CPU memory is not sufficient and the hard disk needs to be used, in order to reduce the GPU's access to the hard disk resources, the tensors in the CPU are dynamically maintained. Specifically, the related tensors of a fixed number of layers L are dynamically maintained in the limited CPU memory. When the calculation reaches the i-th layer, the GPU pre-fetches the related tensors of the i+1-th layer, and the CPU also pre-fetches the related tensors of the i+L-th layer from the hard disk, and removes the related tensors of the i-th layer. In this way, the idle CPU-hard disk transmission bandwidth can be effectively utilized to reduce the GPU's access to the hard disk.

[0065] Step 102: Input the pre-run text data set into the preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, and determine the hybrid expert module calculation transmission time and the expert correlation table.

[0066] The pre-run text data set includes a plurality of pre-run text data.

[0067] It should be noted that the present invention performs expert pre-fetching based on the relevance of expert selection. In the reasoning process of the preset hybrid expert model, there are popular experts, that is, a small number of experts can cover most of the calculations. Therefore, the pre-fetching objects of the MoE layer are gated functions and popular experts. Since MoE is data-sensitive, popular experts may change with changes in input. Therefore, in order to identify the popular experts that tokens in the current batches tend to select, the present invention establishes a data-aware expert relevance table.

[0068] Specifically, the correlation (i.e., frequency relationship) between the experts activated at different layers of tokens (pre-run text data) is recorded through the pre-run process, thereby obtaining an expert correlation table. At the same time, in order to perceive the hardware capabilities in the current environment. Before using the preset hybrid expert model for reasoning in the pre-run process, the present invention measures the time for calculating batches of different sizes in the current environment based on the shape, dtype and other information of different layers of the model, as well as their transmission time, thereby obtaining the hybrid expert module calculation transmission time, and recording the result (hybrid expert module calculation transmission time) locally as a cache.

[0069] Step 103: Based on the expert correlation table and the preset number of pre-fetched hot experts, hot expert identification is performed on multiple hybrid expert modules in the preset hybrid expert model to determine multiple target hybrid expert modules.

[0070] Specifically, step 103 may include the following sub-steps S31-S34:

[0071] Step S31, based on the expert correlation table, determining the expert selection frequencies of the plurality of pre-operation text data in the pre-operation text data set for the plurality of experts in each hybrid expert module;

[0072] Step S32, sorting the expert selection frequencies of each expert in each mixed expert module for each pre-running text data in descending order;

[0073] Step S33: among the experts in each mixed expert module, experts corresponding to the expert selection frequency of the pre-set number of hot experts are selected as hot experts, and the remaining experts are selected as non-hot experts;

[0074] Step S34: Use the mixed expert module corresponding to each popular expert as the target mixed expert module.

[0075] It should be noted that, in reasoning, a table lookup is performed based on the expert selection of the first l layers to determine the expert tendency of each token in the current layer. For example, assuming that there are four experts in the expert layer in each hybrid expert module, gate selects top-1, l=1, and the larger l is, the more accurate the pre-fetching is. For the expert activation path of each token in multi-batch, the present invention looks up the table to determine the expert selection tendency (expert selection frequency) of each token in the current layer, and then aggregates the tendency of all tokens in the multi-batch. In each hybrid expert module, the top-K (pre-set number of popular experts to be pre-fetched) experts are selected for pre-fetching, that is, the experts corresponding to the expert selection frequency of the preset number of popular experts to be pre-fetched are selected as popular experts, and the remaining experts are selected as non-popular experts; among them, these K experts will cover the calculation of most tokens.

[0076] Exemplarily, assuming that the expert layer in each hybrid expert module includes four experts, and the preset number of popular experts is set to 1, then after sorting the expert selection frequency of each expert in each hybrid expert module in descending order, the expert corresponding to the previous expert selection frequency is selected as the popular expert of the corresponding hybrid expert module, and the remaining three experts are selected as non-popular experts. The hybrid expert module corresponding to each popular expert is used as the target hybrid expert module. At this time, the expert layer in the target hybrid expert module includes one popular expert and three non-popular experts.

[0077] In this embodiment, the establishment of the expert correlation table is completed by pre-running. The present invention dynamically updates the correlation table during the reasoning process so that it can better understand the data tendency of the current task as the reasoning proceeds. However, the update is not saved locally to prevent the data tendency in other tasks from affecting this task. For non-expert tensors, the present invention adopts the same method as the unloading strategy for dense models, that is, pre-fetching the tensor when performing the previous layer of calculation. Because non-expert tensors only participate in calculations once in a forward process, and are inactive at other times.

[0078] Step 104: Use preset constraint-aware planning inequalities to calculate the transmission time based on the preset number of pre-fetched popular experts and the hybrid expert module to determine the number of input text batches.

[0079] The calculation transmission time of the hybrid expert module includes the attention layer calculation time, the gating layer calculation time, the popular expert calculation time, the average calculation time of a single expert, the attention layer transmission time, the gating layer transmission time, and the single expert transmission time.

[0080] It should be noted that in order to ensure that multiple calculations of the current layer can completely overlap the transmission of the next layer, it is necessary to take into account the batch size and the number n contained in the batch group (the number of input text batches), the hardware capabilities in the environment, that is, the transmission bandwidth between heterogeneous memories and the speed of GPU calculation. However, the hardware capabilities in the environment are fixed, and the batch size is generally a multiple of 4, so the space for selection is not large. If n is too large, a large amount of KV cache will be introduced; if n is too small, the total time of n batch calculations will hardly overlap the time of the next layer of transmission.

[0081] For further information, see Figure 2 In order to explore the value of n, the present invention Figure 2Some arrows are inserted in , and the positions they indicate are the positions where a certain tensor needs to start calculation. Among them, (1) means that the gated calculation will start. (2) means that the calculation of the hot experts will start. (3) represents that the calculation of the non-hot experts will start. (4) indicates that the calculation of the next layer of attention will be carried out. This means that the transmission of the corresponding tensor needs to be completed before this position, indicating that the communication and calculation have been completely overlapped. Therefore, the inequalities that need to be satisfied at the four key positions can be obtained, and the preset constraint-aware planning inequality is obtained. The preset constraint-aware planning inequality can be expressed as:

[0082] ;

[0083] Where n is the number of input text batches, and n is the smallest integer that satisfies the preset constraint-aware planning inequality; Calculate time for attention layer; Transmission time for the gating layer; Calculate time for the gating layer; Transmit time for a single expert; is the calculation time of popular experts; K is the number of preset and pre-fetched popular experts; is the average computation time of a single expert for the i-th expert; is a built-in function; Q is the number of experts; Transfer time for the attention layer.

[0084] It is worth mentioning that the transmission time and calculation time will vary with the hardware environment, model and batch size. The length of the expert queue Q activated at each layer will not be fixed.

[0085] Furthermore, the hybrid expert module obtained based on the above steps calculates the transmission time, solves the preset constraint-aware planning inequality, and obtains the value of the number of input text batches n. Assuming that the final result is n>=x, then n= [x]. Because n at this time has satisfied a bubble-free pipeline. If n is further increased, the throughput will be improved but the amplitude is low. Because there are no bubbles in the pipeline, the improvement at this time mainly comes from the reduction in the number of IO times. This means that the increase in n can only bring a slight marginal benefit, and the resulting large amount of KV cache will add a greater burden to the storage. Therefore, n only needs to take the smallest integer that satisfies the inequality. In addition, if the value of n is too large, it may be necessary to manually adjust the strategy. Since n is a positive integer, this process is not difficult.

[0086] Step 105: In a multi-level heterogeneous memory space, each target hybrid expert module is used to output a model reasoning result corresponding to the subset of text data to be inferred according to the subset of text data to be inferred corresponding to the number of input text batches.

[0087] It should be noted that after completing the deployment of the model (construction of multi-level heterogeneous memory space) and the generation of the expert correlation table, in order to provide efficient services, the present invention designs a tensor scheduling algorithm based on popular experts, and realizes efficient reasoning in the current environment by perceiving and solving hardware constraints.

[0088] For further information, see Figure 3 , the multi-batch calculation diagram of the MoE model is as follows Figure 3 As shown in the figure, each row is a request batch, and the traditional computing method completes the calculation of each batch row by row. However, in a resource-constrained environment, most tensors are unloaded, and the row-by-row calculation method is difficult to ensure complete overlap of communication and calculation (that is, the transmission of the next layer of tensors is completed within the time period for the current layer to be calculated).

[0089] In a multi-batch scenario, after a model loads a certain layer, it performs multiple vertical calculations and loads the weights of the next layer at the same time, so that the total time of multiple calculations covers the time of loading a layer, reducing the average latency and IO times. However, the MoE layer contains gating / routing functions and multiple experts, which are large in size and slow in transmission, which may lead to long waiting times, and not all experts participate in the calculation, resulting in communication waste.

[0090] Therefore, the present invention proposes to perform column-by-column calculation, that is, to consider the calculation of multiple batches (a batch group) at the same time to achieve weight sharing.

[0091] Optionally, the target hybrid expert module includes a gating layer, an expert layer and an attention layer. The expert layer includes a hot expert with a preset number of hot experts and a plurality of non-hot experts; the data processing process of the target hybrid expert module can be implemented by executing the following steps S01 to S08:

[0092] Step S01, weighting multiple input text unit data in the input text data subset input to the target hybrid expert module through the gating layer to determine the weight value corresponding to each non-hot expert for each input text unit data;

[0093] Step S02, sorting the weight values ​​corresponding to any input text unit data of each non-hot expert in descending order, and selecting non-hot experts corresponding to a preset number of weight values ​​as non-hot activated experts;

[0094] Step S03, counting the non-hot activation experts corresponding to each input text unit data, and determining the activation frequency corresponding to each non-hot activation expert;

[0095] Step S04, sorting the activation frequencies corresponding to the non-hot activation experts in descending order, and generating a non-hot activation expert data processing sorting result;

[0096] Step S05, using the hot experts with a preset number of hot experts to respectively calculate the input text data subset, and outputting the hot expert calculation results corresponding to the hot experts with the preset number of hot experts;

[0097] Step S06, using the input text data subsets in sequence as inputs of non-hot activation experts corresponding to the non-hot activation expert data processing and sorting results, and outputting multiple non-hot activation expert calculation results;

[0098] Step S07, merging the hot expert calculation results and the non-hot activated expert calculation results to generate a merged expert calculation result;

[0099] Step S08: The preset attention mechanism is used to calculate the combined expert calculation results through the attention layer to generate an output model result.

[0100] It should be noted that the present invention splits the MoE layer into the gating / routing function and the expert. When calculating the previous layer, the weights of the gating / routing function and the popular expert are pre-fetched. When calculating the gating / routing function, it is determined whether the selected expert is a popular expert or a selected expert. If not, the weight of the expert is transmitted. Figure 3 As shown, since experts 2 and 4 are popular experts and have completed pre-fetching, while experts 5 and 3 are still transmitting, the calculation order of [2523424…] will cause the GPU to be blocked at positions 5 and 3, because at this time they are most likely not yet able to complete the transmission. However, the subsequent calculations of 2 and 4 could have been performed directly. Therefore, in order to reduce this unnecessary delay, the present invention further adjusts the order of expert calculations involved in multiple batches, merges the calculations of the same experts, and gives priority to the calculations of popular experts. Because popular experts are not only transmitted to the GPU memory the earliest, but also participate in the largest number of calculations. This can provide more sufficient time for subsequent experts who have not completed the transmission. After completing the calculation of the popular experts, the remaining experts perform calculations in the order of transmission. In addition, experts who have completed all calculations will be unloaded in advance, rather than waiting until the entire layer of calculations is completed to unload, in order to reduce the peak value of the video memory.

[0101] Specifically, when performing calculations at the attention layer and the gating layer, the present invention asynchronously transmits popular experts to achieve overlap between layers. When the gating results (activated popular experts, non-popular activated experts) are obtained, when entering the expert layer calculation, the calculation of the popular experts is prioritized (the transmission has been completed, no need to wait), and the remaining experts (non-popular activated experts) are asynchronously transmitted in descending order, achieving overlap between experts (i.e., overlap within the layer). After the popular experts complete the calculation, the popular expert results are obtained. Then, according to the sorting results of the non-popular activated expert data processing, that is, the non-popular activated experts process the input data in the order in which the input data is processed in the results, and obtain the corresponding non-popular activated expert calculation results; for example, assuming the preset number is 2, if the expert layer in the target mixed expert module includes one popular expert and three non-popular experts (non-popular expert A, non-popular expert B, non-popular expert C), based on the gating layer for each input text in the input text data subset After calculation of this data, the weight value corresponding to each non-popular expert for each input text unit data is obtained, the weight values ​​corresponding to each non-popular expert for any input text unit data are sorted in descending order, and the non-popular experts corresponding to the first two weight values ​​are selected as non-popular activation experts (non-popular activation expert A, non-popular activation expert B), and corresponding statistics are made to obtain the activation frequency corresponding to each non-popular activation expert. After sorting, the non-popular activation expert data processing sorting result for non-popular activation expert A and non-popular activation expert B is: non-popular activation expert B-non-popular activation expert A (that is, non-popular activation expert B first completes the calculation of the input data, and then calculates with non-popular activation expert A); wherein, the number of preset pre-fetched hot experts can be set to 1 or 2, and the preset number can be set as needed, and the present invention is not limited to this.

[0102] For further information, see Figure 4 , the final calculation of the target hybrid expert module is as follows Figure 4 The computation graph shown in the figure performs calculations and shares the loaded weights among multiple batches, which not only reduces the number of IOs to about 1 / n of the original number, but also overlaps the time of each IO to achieve a pipeline with almost no bubbles, greatly improving the throughput of model inference.

[0103] It is worth mentioning that, assuming that the number of layers of the preset hybrid expert model is 24, that is, the model contains 24 MoEBlocks (hybrid expert modules), it means that the preset hybrid expert model will include 24 target hybrid expert modules. Each target hybrid expert module calculates and processes the input data based on the step principles of the above target hybrid expert module. That is, after the whole process is cycled multiple times (that is, the number of model layers), the output result is the model inference result. The number of output model inference results corresponds to the number of text data subsets to be inferred corresponding to the number of input text batches. Assuming that the number of input text batches is 3, it means that the number of input text data subsets to be inferred is 3, and the number of output model inference results is also 3.

[0104] As a comparison of technical effects, we can refer to existing technologies. In recent years, with the rapid development of deep learning, large models have achieved remarkable results in various fields. In order to scale the model without increasing the cost of training and inference, recent work has introduced a sparsely activated mixed expert model, namely Mixture-of-Experts (MoE). The MoE model usually uses the MoE layer to replace the feed-forward neural network layer (FFN). For each input, it only requires sparse activation of some parameters (i.e., experts) to participate in the calculation, which greatly reduces the computational cost. The current work has proved the superiority of the MoE structure through a large number of experiments, but the MoE model faces a more severe memory bottleneck challenge than other large models in the inference stage, which is mainly due to the huge parameter scale of MoE. For example, DeepSeek-V2 has 236B parameters, which requires at least eight of the most advanced GPUs (such as A100) to perform inference. This challenge is particularly serious when resources are limited.

[0105] To address the memory optimization problem during large model reasoning, offloading is one of the current mainstream solutions. Tensors that are not currently involved in the calculation can be offloaded to the CPU memory or hard disk to significantly reduce the demand for video memory. However, due to the physical condition that the current computing speed is generally faster than the transmission speed, it is difficult for existing technologies to achieve complete overlap between computing and communication, resulting in a large number of "bubbles" in the reasoning pipeline, resulting in low reasoning efficiency.

[0106] Among them, see Figure 6 ,In recent years, with the rapid development of deep learning, large models have gradually penetrated into various fields.,The existing large models are mainly based on the Transformer structure, such as Figure 6(a) Since GShard introduced the novel mixed expert structure, researchers have continuously explored its huge potential in improving the performance of large language models, and it has gradually become one of the mainstream structures of large language models. Popular models such as GPT-4, Gemini 1.5, and Mixtral 8x7B all use the MoE structure. Figure 6 Middle: Transformer block structure and MoE block structure. The Transformer-based large model and the MoE-based large model are respectively composed of multiple Transformer blocks and MoE blocks connected. E represents expert.

[0107] Unlike the large model based on the Transformer structure, the hybrid expert model uses the MoE layer to replace the feedforward neural network layer, such as Figure 6 (b) As shown. The MoE layer consists of a gating network and multiple experts. The gating function is the key feature of MoE, which discretely selects the experts involved in the calculation for each input. The most common practice is to select the top k experts for each token. In this case, the gating function is usually a softmax function, which activates the top-k experts according to the calculated probability distribution, and k is generally 1 or 2. Existing research shows that the expert selection path of each token can indicate the characteristics of the token and can be used to predict the expert selection.

[0108] Experts are essentially feed-forward neural networks that are sparsely activated based on the results of the gating function. It is this sparse activation property that makes it the most feasible approach to obtain larger models. For each input, the final output of the MoE layer is a weighted sum of the outputs from the selected experts.

[0109] Furthermore, one of the current mainstream solutions is to optimize the graphics memory during large model reasoning. During the model reasoning process, each tensor does not participate in the calculation throughout the process. By unloading the tensors that are not currently involved in the calculation to the CPU memory or hard disk, a large amount of graphics memory space can be released. So in theory, as long as the graphics memory is sufficient to hold the computing resources, and the CPU memory and hard disk are sufficient to unload other resources, model reasoning can be performed. However, in contrast, the offloading strategy is always accompanied by a delay problem. Because the relevant tensors are required to exist in the GPU graphics memory during calculation, the speed of tensor transmission is often less than the speed of GPU calculation due to the limitation of transmission bandwidth. The GPU can only wait for the tensor transmission to be completed before performing calculations, resulting in a waste of computing power.

[0110] In summary, the existing model reasoning process is mainly aimed at traditional dense models. When it is directly applied to MoE models, there will be performance loss and reduced reasoning efficiency. This is mainly because they pre-fetch the next layer indiscriminately when performing current layer calculations, without considering the differences in the demand for IO resources of different layers. When this approach is applied to dense models, it is efficient because there is only one FFN in each Transformer block. However, there are multiple FFNs (i.e., multiple experts) in the MoE block. Directly pre-fetching the entire MoE layer will cause a large number of "bubbles" in the pipeline. At the same time, due to the physical conditions that the current computing speed is generally faster than the transmission, the IO of a single expert is much larger than the computing time. Many existing works focus on improving the accuracy of pre-fetching experts, but even if 100% accurate pre-fetching is achieved, there are still a large number of bubbles in the pipeline because the IO of the expert is long. When it is applied to models with a large number of expert selections k, the number of "bubbles" increases exponentially, and the delay will be more significant.

[0111] In view of the above problems, the present invention proposes a text data reasoning method based on a hybrid expert model to achieve high-throughput MoE reasoning in resource-constrained scenarios. The present invention first analyzes the reasons for the inefficiency of the current MoE reasoning pipeline, and then proposes a multi-batch processing pipeline based on expert perception, which uses the high computing and low IO characteristics of popular experts to arrange multiple batches of calculations to minimize inter-layer bubbles and intra-layer bubbles in the pipeline. In addition, the present invention also adaptively measures the hardware capabilities in the current environment to generate the optimal execution plan in the current environment.

[0112] In an embodiment of the present invention, the present invention provides a text data reasoning method based on a hybrid expert model. When it is necessary to reason about text data, a multi-level heterogeneous memory space is first constructed, and a pre-run text data set is obtained; then, the pre-run text data set is input into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, and the hybrid expert module calculates the transmission time and the expert correlation table to determine; based on the expert correlation table and the preset number of pre-fetched hot experts, hot expert identification is performed on multiple hybrid expert modules in the preset hybrid expert model to determine multiple target hybrid expert modules; a preset constraint-aware planning inequality is used to calculate the transmission time according to the preset number of pre-fetched hot experts and the hybrid expert module to determine the number of input text batches; finally, in the multi-level heterogeneous memory space, the pre-run text data set is input into a preset hybrid expert model for pre-run reasoning, and the hybrid expert module calculates the transmission time and the expert correlation table to determine the number of target hybrid expert modules; based on the expert correlation table and the preset number of pre-fetched hot experts, the hybrid expert module calculates the transmission time and the number of input text batches; finally, in the multi-level heterogeneous memory space, the pre-run text data set is input into a preset hybrid expert model for pre-run reasoning, and ... In the storage space, each target hybrid expert module is used to output the model reasoning result corresponding to the text data subset to be reasoned according to the input text batch number. Based on the above scheme, according to the pre-running reasoning process, the hardware capabilities in the current environment are adaptively measured, the hybrid expert module calculation transmission time and the expert correlation table are determined, and the preset constraint perception planning inequality is combined to output the input text batch number, and the text data subset to be reasoned corresponding to the input text batch number is used as the input of the multiple target hybrid expert modules determined based on the expert correlation table for popular expert identification, and the model reasoning result is output. The present invention focuses on the currently popular hybrid expert model, and greatly compresses the "bubbles" existing in the reasoning pipeline, thereby improving the reasoning efficiency.

[0113] For better explanation, refer to Figure 5 , showing a flow chart of a text data reasoning method based on a hybrid expert model provided in the second embodiment of the present invention. It should be pointed out that this embodiment only briefly describes the general process of the text data reasoning method based on a hybrid expert model, and the specific implementation process of each step can be understood by referring to the relevant content in the aforementioned embodiment, which will not be described here. It can be understood that the present invention is not limited to this.

[0114] The general process of the text data reasoning method based on the hybrid expert model proposed in the present invention is divided into two parts: a planning stage and an execution pipeline stage; the planning stage specifically includes: an adaptive model deployment module: constructing a multi-level heterogeneous memory space, including GPU video memory, CPU memory and hard disk; an expert correlation table generation module: inputting a pre-run text data set into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, and determining the hybrid expert module calculation transmission time and the expert correlation table; an IO-computation pipeline planning module: combining hardware constraint perception and a tensor scheduling algorithm based on hot experts, that is, using preset constraint perception planning inequalities to calculate the transmission time according to the preset pre-fetched hot expert number and the hybrid expert module, and determining the number of input text batches; performing hot expert identification based on the expert correlation table and the preset pre-fetched hot expert number, and determining multiple target hybrid expert modules; in the execution pipeline stage, the pipeline is executed according to the calculation sequence arranged by the present invention and the input text batch number n obtained by solving, and the model reasoning result is output. Specifically, it includes: in a multi-level heterogeneous memory space, each target mixed expert module is used to output a model reasoning result corresponding to the text data subset to be reasoned according to the text data subset to be reasoned corresponding to the number of input text batches.

[0115] Among them, the present invention arranges multiple batches of computing processes based on popular experts, and uses the high computing and low IO characteristics of popular experts to arrange multiple batches of computing, so as to minimize the "bubbles" existing in the pipeline and improve the reasoning efficiency. At the same time, the present invention also designs a constraint-aware IO-computation planner (preset constraint-aware planning inequalities) to formulate reasoning pipelines under different environments; at the same time, the present invention focuses on the reasoning efficiency of MoE in resource-constrained scenarios, and realizes high-throughput MoE reasoning in resource-constrained scenarios by designing a reasoning engine for MoE. In addition, the present invention focuses on the reasoning efficiency of the MoE model in resource-constrained scenarios. Compared with the existing technical solutions, the present invention not only focuses on the currently popular MoE model, but also greatly compresses the bubbles existing in the reasoning pipeline, and greatly improves the efficiency of multi-batch processing while realizing model unloading, realizing high-throughput reasoning of the MoE model in a resource-constrained environment.

[0116] See also Figure 7 , Figure 7 This is a structural block diagram of a text data reasoning device based on a hybrid expert model provided in Example 3 of the present invention.

[0117] The present invention provides a text data reasoning device based on a hybrid expert model, comprising:

[0118] A response module 701 is used to respond to an inference request, build a multi-level heterogeneous memory space, and obtain a pre-run text dataset;

[0119] A pre-run reasoning module 702 is used to input the pre-run text data set into a preset hybrid expert model in a multi-level heterogeneous memory space for pre-run reasoning, and determine the hybrid expert module calculation transmission time and the expert correlation table;

[0120] An identification module 703 is used to identify hot experts for multiple hybrid expert modules in a preset hybrid expert model based on the expert correlation table and the preset number of hot experts, and determine multiple target hybrid expert modules;

[0121] The batch quantity determination module 704 is used to determine the number of input text batches by using a preset constraint-aware planning inequality to calculate the transmission time according to the preset number of pre-fetched popular experts and the hybrid expert module;

[0122] The output result module 705 is used to output the model reasoning result corresponding to the text data subset to be inferred according to the text data subset to be inferred corresponding to the number of input text batches in the multi-level heterogeneous memory space by using each target hybrid expert module.

[0123] Further, the identification module 703 is specifically used for:

[0124] Determining, based on the expert relevance table, expert selection frequencies of a plurality of pre-operation text data in the pre-operation text data set for a plurality of experts in each hybrid expert module;

[0125] Sorting the expert selection frequencies of each expert in each mixed expert module for each pre-run text data in descending order;

[0126] Among the experts in each hybrid expert module, the experts whose selection frequencies correspond to the number of pre-set popular experts are selected as popular experts, and the remaining experts are selected as non-popular experts;

[0127] The hybrid expert module corresponding to each popular expert is used as the target hybrid expert module.

[0128] Furthermore, the target hybrid expert module includes a gating layer, an expert layer and an attention layer; the expert layer includes a hot expert with a preset number of hot experts and a plurality of non-hot experts; the data processing process of the target hybrid expert module is specifically as follows:

[0129] The weights of the multiple input text unit data in the input text data subset input to the target hybrid expert module are calculated through the gating layer to determine the weight values ​​corresponding to the input text unit data by each non-popular expert;

[0130] The weight values ​​corresponding to any input text unit data of each non-hot expert are sorted in descending order, and non-hot experts corresponding to a preset number of weight values ​​are selected as non-hot activated experts;

[0131] Counting the non-hot activation experts corresponding to each input text unit data to determine the activation frequency corresponding to each non-hot activation expert;

[0132] Sort the activation frequencies corresponding to the non-hot activation experts in descending order to generate the non-hot activation expert data processing and sorting results;

[0133] Use the hot experts with a preset number of hot experts to calculate the input text data subsets respectively, and output the hot expert calculation results corresponding to the hot experts with the preset number of hot experts;

[0134] Using the input text data subsets in turn as the inputs of the non-hot activation experts corresponding to the non-hot activation expert data processing and sorting results, and outputting multiple non-hot activation expert calculation results;

[0135] Merging the popular expert calculation results and the non-popular activated expert calculation results to generate a merged expert calculation result;

[0136] The preset attention mechanism is used through the attention layer to calculate the combined expert calculation results and generate the output model results.

[0137] Furthermore, the multi-level heterogeneous memory space includes GPU memory, CPU memory and hard disk; GPU memory is used to store tensors required for current calculations and pre-fetched tensors; CPU memory and hard disk are used to store inactive tensors.

[0138] Furthermore, different types of tensors are placed in the GPU memory, CPU memory, and hard disk respectively according to preset layer-by-layer allocation conditions.

[0139] Furthermore, the hybrid expert module calculates the transmission time including the attention layer calculation time, the gating layer calculation time, the popular expert calculation time, the average calculation time of a single expert, the attention layer transmission time, the gating layer transmission time, and the single expert transmission time; the preset constraint-aware planning inequality is specifically:

[0140] ;

[0141] Where n is the number of input text batches, and n is the smallest integer that satisfies the preset constraint-aware planning inequality; Calculate time for attention layer; Transmission time for the gating layer; Calculate time for the gating layer; Transmit time for a single expert; is the calculation time of popular experts; K is the number of preset and pre-fetched popular experts; is the average computation time of a single expert for the i-th expert; is a built-in function; Q is the number of experts; Transfer time for the attention layer.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the text data reasoning method based on the hybrid expert model as described in the first embodiment above.

[0144] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the text data reasoning method based on the hybrid expert model as described in the first embodiment are implemented.

[0145] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the text data reasoning method based on a hybrid expert model as described in the first embodiment above.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text data reasoning method based on a hybrid expert model, characterized in that: include: Respond to inference requests, build multi-level heterogeneous memory spaces, and obtain pre-run text datasets; Inputting the pre-run text data set into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, determining a hybrid expert module calculation transmission time and an expert correlation table; Based on the expert correlation table and the preset number of pre-fetched hot experts, performing hot expert identification on multiple hybrid expert modules in the preset hybrid expert model to determine multiple target hybrid expert modules; The preset constraint-aware planning inequality is used to calculate the transmission time according to the preset number of pre-fetched popular experts and the hybrid expert module to determine the number of input text batches; In the multi-level heterogeneous memory space, each target hybrid expert module is used to output a model reasoning result corresponding to the subset of text data to be inferred according to the subset of text data to be inferred corresponding to the number of input text batches.

2. The text data reasoning method based on the hybrid expert model according to claim 1 is characterized in that: The method of performing hot expert identification on multiple hybrid expert modules in the preset hybrid expert model based on the expert correlation table and the preset number of hot experts to determine multiple target hybrid expert modules includes: Based on the expert correlation table, determining the expert selection frequencies of the plurality of pre-operation text data in the pre-operation text data set for the plurality of experts in each of the hybrid expert modules; Sorting the expert selection frequencies of each of the pre-running text data for each of the experts in each of the hybrid expert modules in descending order; Among the experts in each of the mixed expert modules, experts corresponding to the expert selection frequencies of the pre-set number of popular experts are selected as popular experts, and the remaining experts are selected as non-popular experts; The mixed expert module corresponding to each of the popular experts is used as the target mixed expert module.

3. The text data reasoning method based on the hybrid expert model according to claim 1 is characterized in that: The target hybrid expert module includes a gating layer, an expert layer and an attention layer; the expert layer includes a preset number of hot experts and a plurality of non-hot experts; the data processing process of the target hybrid expert module is specifically as follows: Performing weight calculation on a plurality of input text unit data in the input text data subset input to the target hybrid expert module through a gating layer, and determining a weight value corresponding to each non-hot expert for each input text unit data; The weight values ​​corresponding to any input text unit data of each of the non-hot experts are sorted in descending order, and non-hot experts corresponding to a preset number of weight values ​​are selected as non-hot activated experts; Performing statistics on the non-hot activation experts corresponding to each of the input text unit data to determine the activation frequency corresponding to each of the non-hot activation experts; Sorting the activation frequencies corresponding to the non-hot activation experts in descending order to generate a non-hot activation expert data processing and sorting result; Use a preset number of hot experts to calculate the input text data subset respectively, and output the hot expert calculation results corresponding to the preset number of hot experts; Using the input text data subsets in sequence as inputs of non-hot activation experts corresponding to the non-hot activation expert data processing and sorting results, and outputting a plurality of non-hot activation expert calculation results; Merging the hot expert calculation results and the non-hot activated expert calculation results to generate a merged expert calculation result; The combined expert calculation results are calculated by the attention layer using a preset attention mechanism to generate an output model result.

4. The text data reasoning method based on the hybrid expert model according to claim 1 is characterized in that: The multi-level heterogeneous memory space includes GPU memory, CPU memory and hard disk; the GPU memory is used to store tensors required for current calculation and pre-fetched tensors; the CPU memory and the hard disk are used to store unactivated tensors.

5. The text data reasoning method based on the hybrid expert model according to claim 4 is characterized in that: According to preset layer-by-layer allocation conditions, different types of tensors are placed in the GPU video memory, the CPU memory and the hard disk respectively.

6. The text data reasoning method based on hybrid expert model according to claim 1, characterized in that: The hybrid expert module calculates the transmission time including the attention layer calculation time, the gating layer calculation time, the popular expert calculation time, the average calculation time of a single expert, the attention layer transmission time, the gating layer transmission time, and the single expert transmission time; the preset constraint-aware planning inequality is specifically: ； Where n is the number of input text batches, and n is the smallest integer that satisfies the preset constraint-aware planning inequality; Calculate time for attention layer; Transmission time for the gating layer; Calculate time for the gating layer; Transmit time for a single expert; is the calculation time for popular experts; K is the number of preset and pre-fetched popular experts; is the average computation time of a single expert for the i-th expert; is a built-in function; Q is the number of experts; Transfer time for the attention layer.

7. A text data reasoning device based on a hybrid expert model, characterized in that: include: The response module is used to respond to inference requests, build multi-level heterogeneous memory spaces, and obtain pre-run text datasets; A pre-run reasoning module, used for inputting the pre-run text data set into a preset hybrid expert model in the multi-level heterogeneous memory space for pre-run reasoning, and determining a hybrid expert module calculation transmission time and an expert correlation table; An identification module, used for performing hot expert identification on a plurality of hybrid expert modules in the preset hybrid expert model based on the expert correlation table and the preset number of hot experts, and determining a plurality of target hybrid expert modules; A batch quantity determination module is used to determine the number of input text batches by using a preset constraint-aware planning inequality according to the preset number of pre-fetched popular experts and the transmission time calculated by the hybrid expert module; The output result module is used to output the model reasoning result corresponding to the text data subset to be inferred according to the text data subset to be inferred corresponding to the number of input text batches in the multi-level heterogeneous memory space by using each of the target hybrid expert modules.

8. A computer device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the text data reasoning method based on the hybrid expert model as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the text data reasoning method based on the hybrid expert model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer is caused to execute the text data reasoning method based on a hybrid expert model as described in any one of claims 1-6.

Images